The goal of this tutorial is to create a search page in which the user can search and filter data stored in a Elastic Search database. We will build such a page using ASP.NET (Core) and the ElasticSearch.Nest library.
We are going to:
- build a HTML Form to talk our backend
- build a ASP.NET controller wich will search the Elastic Search database
- build a Razor View wich will show the result of the search query
- extend our controller whith Aggregations
- extend our view to show the user those aggregations
- turn those aggregations into filters
- and add those filters to our controller
In the end we will have a web application that will look something like this.
Case
In our example we will build a web page in which we can search and filter for publications. A publication will look as follows:
public class PublicationDocument
{
public string Title { get; set; }
public string Content { get; set; }
public List<string> Tags { get; set; }
}
Basic Setup
Of course we need a ASP.NET (Core) web application and a running Elastic Search node. Check this tutorial and this tutorial if you don’t know how to do this.
Nest
First we need to install Nest. Nest is a C# client library which can talk to Elastic Search.
Install-Package Nest
View and Controller
First we need to create a html form which will send our search query to our controller. Our basic form in our index.cshtml file will look like this.
<form class="form-inline" asp-controller="Search" asp-action="Index" method="get">
<div class="form-group">
<label for="query">Query</label>
<input class="form-control" id="query" type="text" name="query" />
</div>
<button class="btn btn-default" type="submit">Search</button>
</form>
When the Search button is clicked. This form will send a Http GET request to the Index method/action of our SearchController. So we will need to create a method/action which can handle this request.
public class SearchController : Controller
{
public IActionResult Index(string query, ICollection<string> filters)
{
return View();
}
}
When the form is submitted, ASP.NET will transform the request into a method call to our Index method. The query parameter will contain the value of the text input. We will use the filter parameter later on.
Search
Nest Client
In order to talk to the Elastic Search database with Nest, we need to create a ElasticSearchClient. I won’t go to deep into how to do this, but basically you create a new client object. We also need to create a index for our PublicationDocument and map the properties.
var client = new ElasticClient(new Uri("http://localhost:9200"));
client.CreateIndex(indexName, index => index
.Mappings(mappings => mappings
.Map<PublicationDocument>(mapping => mapping.AutoMap())));
Query
We will start with just a simple string query. Later on we are going to extend this query with aggregations and filters. But for now our query will look like this. In this example, the query variable is the parameter passed to our method/action.
public IActionResult Index(string query, ICollection<string> filters)
{
var search = new SearchDescriptor<PublicationDocument>()
.Query(qu => qu
.QueryString(queryString => queryString
.Query(query)));
var result = client.Search<PublicationDocument>(search);
}
Our result will contain objects that will represent the following JSON object. Nest has already converted this JSON object to C# objects.
{
"took" : 100,
"timed_out" : false,
"_shards" : { ... },
"hits" : {
"total" : 24,
"max_score" : 0.22365987,
"hits" : [ {
"_index" : "publications",
"_type" : "publicationdocument",
"_id" : "AVe4A-YkUrk2nBS_otdC",
"_score" : 0.22365987,
"_source" : {
"title" : "Test 4",
"content" : "Quamquam in hac divisione rem ipsam prorsus probo, elegantiam desidero.",
"tags" : [ "Lorum", "Quae", "Similitudo" ]
}
}, ... ]
}
}
DTO
Now we need to pass the hits inside our result of our executed query back to our view. We will do this with so called Data Transfer Object, DTO’s. This are simple POCO’s that will transfer data from our controller to our view. The DTO for this method/action will look as follows.
public class PublicationsResultDTO
{
public List<PublicationDocument> Results { get; set; }
}
For now it will only contain the publications found by our query. Later on we will extend this DTO to also contain the aggregations.
To pass the result back to the view we need to transform the var result into the DTO and return this to our view. In C# we can easily extract all the PublicationDocuments by selecting the source from each hit.
public IActionResult Index(string query, ICollection<string> filters)
{
...
return View(new PublicationsResultDTO()
{
Results = result.Hits
.Select(hit => hit.Source)
.ToList()
});
}
Razor view
In our index.cshtml view we need to display the result of our search request. We will use the DTO returned by our method/action and a razor foreach loop in order to display all the results. This code will create a row for every result found by our query.
@model ElasticSearchFilter.Models.DTO.PublicationsResultDTO
@foreach (var result in Model.Results)
{
<div class="row">
<div class="col-md-12">
<h2>@result.Title</h2>
<p>@result.Content</p>
</div>
</div>
}
Aggregations
Our PublicationDocument contains tags. It would be nice if the user can filter on those tags. In order to do so we first need to aggregate the tags when querying the database. Later on we will convert those tags into filters.
Extend query with aggregations
In order to add the aggregations to our query we need to extend our query. We will aggregate on the Tags field of our PublicationDocument. Now the query becomes:
public IActionResult Index(string query, ICollection<string> filters)
{
var search = new SearchDescriptor<PublicationDocument>()
.Query(qu => qu
.QueryString(queryString => queryString
.Query(query)))
.Aggregations(ag => ag
.Terms("tags", term => term
.Field(field => field.Tags)));
...
}
This will add an aggregation named tags to our query which will aggregate all the tags into Buckets. When executing the search query we will get something back like this.
{
"took" : 100,
"timed_out" : false,
"_shards" : { ... },
"hits" : { ... },
"aggregations" : {
"tags" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" : [ {
"key" : "lorum",
"doc_count" : 24
}, {
"key" : "ipsum",
"doc_count" : 12
}, ... ]
}
}
}
As you can see all tags are grouped into buckets. Each bucket has a Key (the name of the tag) and a count which will tell us how many documents contain those tags. E.g. I have 24 document containing the lorum tag.
Extend the DTO with aggregations
In order to let the user see those aggregations, we need to pass them to our view. We do this by adding an aggregations list to our publicationResultDTO. Our DTO’s will look like this.
public class PublicationsResultDTO
{
public List<PublicationDocument> Results { get; set; }
public AggregationDTO Aggregation { get; set; }
}
public class AggregationDTO
{
public string Name { get; set; }
public List<BucketDTO> Buckets { get; set; }
}
public class BucketDTO
{
public string Key { get; set; }
public string Count { get; set; }
}
Now we need to get the aggregation from the result of our query and convert it to our DTO’s.
public IActionResult Index(string query, ICollection<string> filters)
{
...
return View(new PublicationsResultDTO()
{
Results = ...,
Aggregation = new AggregationDTO()
{
Name = "tags",
Buckets = result.Aggs
.Terms("tags")
.Buckets
.Select(bucket => new BucketDTO() { Key = bucket.Key, Count = bucket.DocCount })
.ToList()
}
}
}
This will create a new AggregationDTO with the name Tags. It will also create a BucketDTO for every bucket contained in the result. The key of the bucket will contain the tag itself, and the Count property of the bucket will contain the document count.
Show the aggregations
In our view we need to show the buckets inside our DTO to the user.
<form asp-controller="Search" asp-action="Index" method="get">
<input type="hidden" name="query" value="@Context.Request.Query["query"]" />
@foreach (var bucket in Model.Aggregation.Buckets)
{
<div class="checkbox">
<label>
<input type="checkbox" name="filters" value="@bucket.Key" @(Context.Request.Query["filters"].Contains(bucket.Key) ? "checked" : "") /> @bucket.Key <span class="badge">@bucket.Count</span>
</label>
</div>
}
<button class="btn btn-default" type="submit">Filter</button>
</form>
As you can see we created a form which contains a checkbox for every bucket. By setting the name of every checkbox to filters, our controller will get all checked values as a ICollection parameter. In order to remember the search query and checked checkboxes across page refreshed, we need to put to previous request data back into to form. So our hidden input will contain the search query originally entered by the user and our checkboxes will be checked if the user has checked them in the previous request. If we submit the form again, no data will be lost.
Filters
Now the user can search our elastic search database and see the aggregated buckets. Also, the user can check a filter and send this to our endpoint. But until now we did not use those filters. In order to use those filters we need to add them to our query.
First we need to transform our filter string to elastic search filter queries. We do this by creating a Term filter on the Tags field of our PublicationDocument for every filter passed to our method/action. This will result into a list of Term filters which we can add to our query.
public IActionResult Index(string query, ICollection<string> filters)
{
var tagsFilter = filters.Select(value =>
{
Func<QueryContainerDescriptor<PublicationDocument>, QueryContainer> tagFilter = filter => filter
.Term(term => term
.Field(field => field.Tags)
.Value(value));
return tagFilter;
});
...
}
Now we add those tags filters to our query. Those filters need to be combined with the query itself. A ‘normal’ query can’t contain both a filter and a string query. In order to use both we need to convert our query into a boolean query. A boolean query can contain multiple queries. The must query will be our string query. And of course, our filters will be our filter query.
public IActionResult Index(string query, ICollection<string> filters)
{
var tagsFilter = ...
var search = new SearchDescriptor<PublicationDocument>()
.Query(qu => qu
.Bool(b => b
.Filter(tagsFilter)
.Must(must => must
.QueryString(queryString => queryString
.Query(query)))))
.Aggregations(ag => ag
.Terms("tags", term => term
.Field(field => field.Tags)));
...
}
Conclusion
We created a search -> aggregate -> filter loop using ASP.NET and Nest. With this loop the user can send a query to our database and filter the result of this query based on the aggregations we created.
Comments