The difficult part of SDN Neo4j

[bytor99999]

OK, I have spent a lot of time on Neo4J and SDN. My app isn't a difficult app and should have been finished weeks ago. I needed Pagination, but that can't be implemented with the Repos with Pageable (from my previous post). I needed to be able to write queries that were easy to understand and get exactly the data that I need, but the docs don't beyond the very simple types of queries. If there was lazy loading, then I might get around the cypher language to figure out how to write the query.

But when you want certain data for a use case you typically retrieve exactly the amount of data you need for the entire use case so that when you get to a UI all the data is there ready for it to display it.

However, I can't seem to get just that. I could run many simple queries within my use case and loop through objects and calculate whatever else I need. But that is piece-mealing it. I should be able to go to the db once or twice in a use case and get the data.

It is really difficult to figure out what was loaded and what wasn't in your query as your code traverses through nodes and relationships. I thought I had it licked in my one use case that I posted in another thread, but my Event had its associated lists populated, but those vertices/nodes didn't have their associations populated even thought they are mapped with @Fetch. There was an object, but all their properties were set to null, even though there is actual data. I got the Event object from the repos findOne() method.

I of course can never give up, that isn't in my nature.
Thanks for all your help.

Mark

[MichaelHunger]

Mark,

thanks a lot for your feedback and all the contributions so far.

Let me answer some of your questions.

* Paging is not so easy as you would have to either write two queries or have to execute the query a second time for just counting the objects (both options might be implemented for 2.1)
* I think that cypher is the best way to specify what you want to have loaded for a use-case, @Fetch handling is still very basic, you're right, what kinds of concrete requirements would you have there (e.g. specify fetch-depth on load or in the annotation etc.?)
* What kind of have you written in the end, can we do something to help you get it done as you've imagined it?

Thanks a lot again

Michael

[bytor99999]

First, I want to apologize. In no means do I want to belittle the amazing job you have done Michael. Spring Data Neo4j and the work with Neo4j you have accomplished goes way beyond what most developers could achieve.

I had to rethink and refactor. I got template.fetch to work and I had to do a couple/few repository calls to get all the data that I needed because the Cypher Query was going to be more complicated than my knowledge of Cypher would allow me to write.

It wasn't so much a fetch depth as Traversing many different relationships from the root node. And then have to do some comparisons.

Here is what I ended up doing. I had 4 of these I needed. Each different, but this is a good example of it. The EventListItem is a UI object for my page, instead of returning the actual mapped domain objects to the UI. It actually made my UI template code a lot easier. And once I did one of these the other 3 were the same pattern, so easier to write. In the end I get exactly what I needed

     @Override
    public ListOfDomains<EventListItem> getListOfEventsForUser() {
        User currentUser = userRepository.getUserFromSession();
        User populatedUser = userRepository.findOne(currentUser.getNodeId());
        template.fetch(populatedUser.getEventsIAmHosting());
        template.fetch(populatedUser.getEventsIAmInvitedTo());
        template.fetch(populatedUser.getEventsIAmAttending());
        ListOfDomains<EventListItem> listOfDomains = new ListOfDomains<EventListItem>();
        calculateListOfEvents(populatedUser, listOfDomains);
        return listOfDomains;
    }
    
    private void calculateListOfEvents(User populatedUser, ListOfDomains<EventListItem> listOfDomains) {
        List<EventListItem> itemsHolder = new ArrayList<EventListItem>();
        for (Event event: populatedUser.getEventsIAmHosting()) {           
            itemsHolder.add(setEventInformationInListItem(event, User.HOSTING));
        }
        for (Event event: populatedUser.getEventsIAmInvitedTo()) {
            itemsHolder.add(setEventInformationInListItem(event, User.INVITED));
        }
        for (Event event: populatedUser.getEventsIAmAttending()) {
            itemsHolder.add(setEventInformationInListItem(event, User.ATTENDING));
        }
        listOfDomains.setDomainObjects(itemsHolder);
        listOfDomains.setType("Events you are involved with.");
    }
    
    private EventListItem setEventInformationInListItem(Event event, String type) {
        EventListItem listItem = new EventListItem();
        listItem.setEventDate(event.getEventDate());
        listItem.setEventId(event.getNodeId());
        listItem.setEventName(event.getTitle());
        listItem.setEventUserType(type);
        return listItem;
    }

For those fetch calls, I couldn't use @Fetch because it could create circular fetches, and I saw what happens when that happens.

Thanks

Mark

[MichaelHunger]

Mark,

could you explain this:

It wasn't so much a fetch depth as Traversing many different relationships from the root node

Do you mean the graph-root or the entity-root node?

Otherwise the code doesn't look too involved, one could extend template.fetch to take a varargs parameter which would shorten this part. And I would probably extract one more method and make EventList item immutable and pass all parameters in the constructor.

For cypher, I think the query would look like:

start user=node:users(login={login})
match user-[r]->event
where type(r) = "ATTENDING" or type(r) = "INVITED" or type(r) = "HOSTING"
return ID(event), event.eventDate, event.title, type(r)

Then you could use the conversion-facilities that allows you to convert each of the returned rows into a EventListItem (much like the spring RowMappers).

HTH

Michael

[bytor99999]

Thanks, That is the part of Cypher that I haven't learned yet. Mostly because I haven't found any docs or tutorials explaining or showing those types of more complex queries. Not that that should be in the SDN docs, but in the Cypher docs at Neo4J or somewhere else.

With that query though, what objects get returned? Can I hook up the ConversionService to the Result set that gets returned by the query like a RowMapper, (answered already by you above)?

So the part about traversal, I posted the easiest of the four use cases. This one will show how I have to go through many traversals on different relationships all through the same query.

    @Override
    public ListOfDomains<ItemListItem> getListOfItemsForEvent(Long eventId) {
        Event event = eventRepository.findOne(eventId);
        User user = userRepository.getUserFromSession();
        ListOfDomains<ItemListItem> listOfDomains = new ListOfDomains<ItemListItem>();
        calculateListOfItems(event, user, listOfDomains);
        return listOfDomains;
    }
    
    private ListOfDomains<ItemListItem> calculateListOfItems(Event event,
                                                             User user,
                                                             ListOfDomains<ItemListItem> listOfDomains) {
        boolean hostOfEvent = isHostOfEvent(event, user);
        List<ItemListItem> itemsHolder = new ArrayList<ItemListItem>();
        for (ItemNeededForEvent neededItem : event.getItemsNeededForEvent()) {
            ItemListItem listItem = new ItemListItem();
            listItem.setEventId(event.getNodeId());
            listItem.setItemNeededForEventId(neededItem.getNodeId());
            listItem.setQuantity(neededItem.getQuantity());
            Item item = itemRepository.findOne(neededItem.getItem().getNodeId());
            listItem.setItemName(item.getDescription());
            listItem.setEditableByHost(hostOfEvent);
            setBooleanValuesInItemListItem(event, user, item, listItem);
            itemsHolder.add(listItem);
        }
        listOfDomains.setDomainObjects(itemsHolder);
        listOfDomains.setType("Items Needed for Event");
        return listOfDomains;
    }
    
    private void setBooleanValuesInItemListItem(Event event, 
                                                User user,
                                                Item item,
                                                ItemListItem listItem) {
        // These are all null associations, need to get them another way
        template.fetch(event.getItemsSignedUpToEvent());
        
        String usersSignedUp = "";
        for (ItemUserSignedUpToBringToEvent signedUpToBringToEvent : event.getItemsSignedUpToEvent()) {
            if (signedUpToBringToEvent.getUser().equals(user)) {
                listItem.setCurrentUserSignedUp(true);
                listItem.setItemSignedUpId(signedUpToBringToEvent.getNodeId());
            }
            if (signedUpToBringToEvent.getItem().equals(item)) {
                usersSignedUp += signedUpToBringToEvent.getUserName() + ", ";
            }
        }
        if (usersSignedUp.length() > 2) {
            usersSignedUp = usersSignedUp.substring(0, usersSignedUp.length()-2);
        }
        listItem.setUsersSignedUpForItem(usersSignedUp);
    }

    private boolean isHostOfEvent(Event event, User user) {
        return event.getEventHosts().contains(user);
    }

The ListItem objects, about making a constructor and immutable. It really doesn't matter if it is immutable or not in my case, and I tend to avoid making constructors, I usually keep the no args constructor. Because I would end up with a constructor with many parameters that I never remember the order of them. I did try to make them Groovy objects, but I couldn't get GMaven to work in my build, but that is nothing of any concern to me anymore.

Thanks Michael

Mark

[bytor99999]

Converters would only work if for each use case there is one and only one cypher query that can be performed. I think in the two examples I posted it might be possible for one query.

Where as with this one with the boolean values, makes it difficult to write a single cypher query for it.
This one is more of what i mean by complex traversal and comparing objects from one node to two different relationship traversals of different types of nodes and returning a Boolean based on that comparison.

    @Override
    public ListOfDomains<ItemListItem> getListOfItemsForEvent(Long eventId) {
        Event event = eventRepository.findOne(eventId);
        User user = userRepository.getUserFromSession();
        ListOfDomains<ItemListItem> listOfDomains = new ListOfDomains<ItemListItem>();
        calculateListOfItems(event, user, listOfDomains);
        return listOfDomains;
    }
    
    private ListOfDomains<ItemListItem> calculateListOfItems(Event event,
                                                             User user,
                                                             ListOfDomains<ItemListItem> listOfDomains) {
        boolean hostOfEvent = isHostOfEvent(event, user);
        List<ItemListItem> itemsHolder = new ArrayList<ItemListItem>();
        for (ItemNeededForEvent neededItem : event.getItemsNeededForEvent()) {
            ItemListItem listItem = new ItemListItem();
            listItem.setEventId(event.getNodeId());
            listItem.setItemNeededForEventId(neededItem.getNodeId());
            listItem.setQuantity(neededItem.getQuantity());
            template.fetch(neededItem.getItem());
            listItem.setItemName(neededItem.getItemName());
            listItem.setEditableByHost(hostOfEvent);
            setBooleanValuesInItemListItem(event, user, neededItem, listItem);
            itemsHolder.add(listItem);
        }
        listOfDomains.setDomainObjects(itemsHolder);
        listOfDomains.setType("Items Needed for Event");
        return listOfDomains;
    }
    
    private boolean isHostOfEvent(Event event, User user) {
        return event.getEventHosts().contains(user);
    }
    
    private void setBooleanValuesInItemListItem(Event event, 
                                                User user,
                                                ItemNeededForEvent itemNeededForEvent,
                                                ItemListItem listItem) {
        // These are all null associations, need to get them another way
        template.fetch(event.getItemsSignedUpToEvent());
        
        String usersSignedUp = "";
        for (ItemUserSignedUpToBringToEvent signedUpToBringToEvent : event.getItemsSignedUpToEvent()) {
            if (signedUpToBringToEvent.getUser().equals(user)) {
                listItem.setCurrentUserSignedUp(true);
                listItem.setItemSignedUpId(signedUpToBringToEvent.getNodeId());
            }
            if (signedUpToBringToEvent.getItem().equals(itemNeededForEvent.getItem())) {
                usersSignedUp += signedUpToBringToEvent.getUserName() + ", ";
            }
        }
        if (usersSignedUp.length() > 2) {
            usersSignedUp = usersSignedUp.substring(0, usersSignedUp.length()-2);
        }
        listItem.setUsersSignedUpForItem(usersSignedUp);
    }

Thanks

Mark

[MichaelHunger]

Mark,

this is the typical code of converting a more involved and fine granular object network to simpler UI-view-object structures. It could be cleaned up a bit e.g. by moving calculateListOfItems to ListOfDomains, and by ItemListItem being able to hydrate itself from a ItemNeededForEvent.

a cypher query for that could look like (so it is not much less involved than the code you have). It is probably just the complexity that lies in this conversion?).

start user=node({user}), event=node({eventId})
match user-[host?:HOSTS]->event,
event-[:ITEMS_NEEDED]->neededItem-[:ITEM]->item,
event-[:ITEMS_SIGNED_UP_FOR]-> signedUpToBringToEvent-[userSignedUp?:USER]->user
return host!=null as isHost, ID(neededItem) as neededItemId, neededItem.quantity as quantity, item.name as itemName,  
userSignedUp!=null as currentUserSignedUp, ID(signedUpToBringToEvent) as itemSignedUpId

The convert I meant are the

List<ItemListItem> = template.query().to(ItemListItem.class, new ResultConverter<Map<String,Object>, ItemListItem >() {
 ItemListItem convert(Map<String,Object> row) {
    create ItemListItem and set the fields from the row
 }
}).as(List.class);

HTH

Michael

[bytor99999]

OK, I was trying a different approach. When you said converter I thought you meant @MapResult and @ResultColumn. So I did the following method in my repository interface

    @Query("start user=node({0}) " +
            "match user-[r]->event " +
            "where type(r) = " + User.ATTENDING + " or type(r) = " + User.INVITED + " or type(r) = " + User.HOSTING + " " +
            "return ID(event), event.eventDate, event.title, type(r) " +
            "order by event.eventDate desc")
    public Iterable<EventItemForAccountPage> getEventsForAccountPage(User user);

and this is the domain object mapped to that query.

package com.perfectworldprogramming.eventgate.event;

import com.perfectworldprogramming.eventgate.wrappers.JsonShortDateSerializer;
import org.codehaus.jackson.map.annotate.JsonSerialize;
import org.springframework.data.neo4j.annotation.MapResult;
import org.springframework.data.neo4j.annotation.ResultColumn;

import java.util.Date;

@MapResult
public interface EventItemForAccountPage {

    @ResultColumn("event.title")
    public String getEventName();

    @ResultColumn("type(r)")
    public String getEventUserType();

    @ResultColumn("ID(event)")
    public Long getEventId();

    @ResultColumn("event.eventDate")
    @JsonSerialize(using= JsonShortDateSerializer.class)
    public Date getEventDate();
}

However, the query doesn't work, I got an error

java.lang.IllegalArgumentException: object is not an instance of declaring class

That second query looks cool. Definitely something you don't find in the docs. But I can see people having a use case for something relatively similar, maybe not as complex, but still with those possible options to have in a cypher query.

I wish I could try the cypher queries directly on the database, but I am only using the embedded version. I don't have Neo4J server installed. Since I really don't see a need for the server version at this moment.

Thanks

Mark

[bytor99999]

start user=node({user}), event=node({eventId})
match user-[host?:HOSTS]->event,
event-[:ITEMS_NEEDED]->neededItem-[:ITEM]->item,
event-[:ITEMS_SIGNED_UP_FOR]-> signedUpToBringToEvent-[userSignedUp?:USER]->user
return host!=null as isHost, ID(neededItem) as neededItemId, neededItem.quantity as quantity, item.name as itemName,  
userSignedUp!=null as currentUserSignedUp, ID(signedUpToBringToEvent) as itemSignedUpId

Michael

I am confused about that query. In the match portion, say the first line. user-[host?:HOSTS]->event will this match only if the user that is currently logged in to my site, which is the user that I pass to the repo method, is the host of the event, or will this return all the events that the user is hosting?

Kind of the same question about "event-[:ITEMS_SIGNED_UP_FOR]-> signedUpToBringToEvent-[userSignedUp?:USER]->user"

"event" and "user" still represent the event for the repo's eventId param and user as the repo's user param. Or are they just aliases used in the query. Because I look at it and think it will give me the Users that have signed up to bring an item to an event, not a specific item. An event could have 10 items NEEDED FOR EVENT, and the current logged in user might only be signed up to bring 1 or 2 of those items and not the other items NEEDED FOR EVENT. and other users, not the one logged in might be signed up for those other items NEEDED FOR EVENT.

Thanks again, I feel it is close. After these uses cases, I just have forms and updates to write code for. All the other queries are all done.

Mark

[MichaelHunger]

You might have to quote the strings of your relationship-type constants.

@Query("start user=node({0}) " +
            "match user-[r]->event " +
            "where type(r) = \"" + User.ATTENDING + "\" or type(r) = \"" + User.INVITED + "\" or type(r) = "\" + User.HOSTING + "\" " +
            "return ID(event), event.eventDate, event.title, type(r) " +
            "order by event.eventDate desc")
    public Iterable<EventItemForAccountPage> getEventsForAccountPage(User user);

Can you provide the full exception that you're running into?

You can run the neo4j-shell (which comes with server, or see willie wheelers post) which executes cypher queries, or use Neoclipse which is also able to work readonly on stores and can execute cypher as well.