Stream distinct by property but exclude by another property

Question

Setup and question

I have a stream over a simple Java data class like:

class Candidate{
    private Long id;
    private String fullName;
    private String job;
    private String adress;
}

I would like to filter my stream by two properties:

• remove all duplicates by job
• but keep everyone with adress "Italy" regardless of job

---

Example

Consider an example data set like

ID	fullName	JOB	adress
1	Peter Bright	IT Engineer	Italy
2	Patrick Manon	Electronics engineer	Spain
3	Bob Jina	IT Engineer	Suisse
4	Alexander Layo	Security Engineer	UK

or in Java:

Candidate c1 = new Candidate(1,"Peter Bright","IT Engineer","Italy");
Candidate c2 = new Candidate(2,"Patrick Manon","Electronics engineer","Spain");
Candidate c3 = new Candidate(3,"Bob Jina","IT Engineer","Suisse");
Candidate c4 = new Candidate(4,"Alexander Layo","Security Engineer","UK");

Stream<Candidate> candidates = Stream.of(c1, c2, c3, c4);

I would like to filter the stream in a way that the outcome is:

ID	fullName	JOB	ADSRESS
1	Peter Bright	IT Engineer	Italy
2	Patrick Manon	Electronics engineer	Spain
4	Alexander Layo	Security Engineer	UK

Note that Bob Jina got removed since IT Engineer was already there.

In the case of there is duplicates candidates all from Italy we need to keep all of them

Answer 1

You could collect to a map of Candidates, using job as the key and keeping Italian candidates whenever there is a collision:

Collection<Candidate> result = candidates.stream()
    .collect(Collectors.toMap(
        Candidate::getJob,
        Function.identity(),
        (c1, c2) -> "italy".equalsIgnoreCase(c1.getAddress()) ? c1 : c2,
        LinkedHashMap::new))
    .values();

This uses the overload of Collectors.toMap that accepts four arguments.

I'm collecting to a LinkedHashMap to preserve insertion-order.

This solution returns candidates in a Collection, if you need a List, create it from the collection:

List<Candidate> list = new ArrayList<>(result);

Answer 2

As I understand it, all jobs, regardless of job type and country will be kept unless Italy is involved. In that case, for any given job, only Italy's will be retained.

You can do it like so. I chose two stages.

• first build a map keyed by jobs.
• then filter those jobs favoring Italy.

Create the map

Map<String, List<Candidate>> jobs = list.stream()
        .collect(Collectors.groupingBy(Candidate::job));

Now, check all the lists.

• if the list for any job does not contain Italy, use that list in its entirety.
• otherwise, use the list excluding all but Italy.

List<Candidate> favorItaly = jobs.values()
     .stream()
     .map(lst -> {
              List<Candidate> italy = lst
                   .stream()
                   .filter(c -> c.getAddress().equalsIgnoreCase("Italy"))
                   .toList();
              // if italy size is 0, use other, else use italy  
              return italy.size() == 0 ? lst : italy;
      })
     .flatMap(List::stream)
     .toList();

The two stages could have been combined but there is no performance improvement in doing so and this way avoids clutter.

Answer 3

Filter

You can achieve this quite easily by using filter and a custom Predicate that is using a Set under the hood to memorize what candidates it already saw.

Also see this highly related answer Java 8 Distinct by property.

Disclaimer: This solution (like the linked answer) proposes a stateful predicate, which, in general, should be considered bad practice and error-prone for various reasons.

---

Filter by distinct name

First of, lets create a Predicate that only accepts a candidate if the job name has not been seen yet. Therefore, lets just steal the code from the post I linked above:

public static <T> Predicate<T> distinctByKey(Function<? super T, ?> keyExtractor) {
    Set<Object> seen = ConcurrentHashMap.newKeySet();
    return t -> seen.add(keyExtractor.apply(t));
}

We create our predicate by writing distinctByKey(Candidate::getJob).

---

Accept all Italians

The last part is to combine this predicate with a predicate that just accepts anyone from Italy, regardless of the job name:

Predicate<Candidate> acceptItalian = candidate -> candidate.getAddress().equals("Italy");

---

Putting it together

Now lets combine both using Predicate#or and use filter on the stream:

Stream<Candidate> candidates = ...
Stream<Candidate> filtered = candidates.filter(
    acceptItalian.or(distinctByKey(Candidate::getJob))
);

And now you have your filtered stream. Call .toList() on it to get a List, for example.

Note that its important that we put the two predicates together in that order. Otherwise the Italians would pollute the Set and lock out non-Itialians with the same job name. The functionality we rely on here is called short-circuiting, which Predicate#or does make use of (this is a documented feature you can rely on).