Spring Data Graph: Map pairs as node properties?

[ractive]

Hello graphistas

We get json data back from a web app and we'd like to store the relevant parts of it in a @NodeEntity. We are currently following the approach to flatten the deep JSON structure to a flat Map<String, Object>, where the keys are converted to a dotted notation.

{
  "foo" : {
    "bar": 1,
    "baz": 2
  }
}

is converted to a Map like

"foo.bar" => 1
"foo.baz" => 2

in an entity like

@NodeEntity
public class MyEntity {
  private Map<String, Object> propertiesToStore;
  ...
} 

We now would like to store the key-value pairs of this propertiesToStore field directly as node properties. Any ideas how this could be implemented with Spring Data Graph?

Or are there even simpler or alternative approaches to store a JSON structure in a node with Spring Data Graph without using a Map as a field?


Best regards,
James

[MichaelHunger]

Right now there is no native support for storing maps in neo4j. You could register a spring converter that converts it to a json string and back again (or to another serialized form).

There is also the possibility to store the map directly as properties of the node? Perhaps prefixed with "foo." as in your example.

You might want to look into the source code of spring data graph and field-accessor-factories. Those allow you to register custom handlers for fields (like a Map<String, Object>) and handling write and read operations to this field.

It would be really cool if you could try that. If you need any hints just ping me. Probably create a JIRA ticket for that and let's continue to discuss there.

Thanks

Michael

[ractive]

I'll check out the field-accessor-factories you mentioned as this sounds very promising. I'll let you know how I'm proceeding with it.

[MichaelHunger]

Cool, looking forward to your solution. For the storage of the individual properties it might be useful to reuse PropertyFieldAccessor and/or ConvertingNodePropertyFieldAccessor probably just extract their functionality to a separate, reusable class.

[ractive]

I could not find a way to register an own FieldAccessorFactory implementation. I opened this JIRA ticket for that.

Just an idea...
I maybe useful to have a Map<String, Object> property be bound to its own referenced node. So maybe the following...

@NodeEntity
public class Person {
    private String name;

    @RelatedTo(type="has")
    Map<String, Object> address;
}

person.address.put("ZIP", 8000);
person.address.put("City", "Zürich"); 

...could create two nodes:
[Person]---(has)--->[address]

where the person node had a "name" property and the address node has the properties "ZIP" and "City" from the map.

This would keep something of the dynamic nature of neo4j.

The background of this: We must persist a lot of graphical information of entities that is not needed in the business layer, but must only be persisted and reached back to the UI again. This graphical information is part of a json document with quite a complex structure. Now instead of denormalizing this structure and creating many (from the backend point of view) useless entity classes, we would like to put this information dynamically in a flat map. This map would be stored ideally in its own node.

What do you think about such a dynamic component in SDG? Could there be other approaches to store key/values dynamically?

[MichaelHunger]

Hi,

you should be able to "register" FieldAccessorFactories by overriding NodeDelegatingFieldAccessorFactory and override the method createAccessorFactories.

But probably there should be a simpler way to register those, I'm not sure.

Interesting to use @RelatedTo for that, I'm a bit leaning towards a custom annotation? (Or probably none).

I've also thought about the solution you're proposing.

What one could do is to have the default-inlined solution for normal Maps and when using @RelatedTo then it is a separate node.

I'd like to have such a dynamic approach being supported in SDG.

I'm not sure about using a generic Map structure. Probably just using a neo4j Node or a SDG-MapNode or a simple PropertyContainer would be enough? For a Map you have to support all the operations of the Map interface some of which don't fit so well in this context (and it would have to be a managed Map anyway whose content must be reflected in the graph on any change)

Michael

[ractive]

I just hacked together a proof of concept implementation that stores all key/value pairs of a PropertyMap field of a NodeEntity to the underlying neo4j node. PropertyMap is an newly introduced PropertyContainer like interface (see below). A own FieldAccessorFactory handles getting and setting this field.

Some observations:

you should be able to "register" FieldAccessorFactories by overriding NodeDelegatingFieldAccessorFactory and override the method createAccessorFactories.

But probably there should be a simpler way to register those, I'm not sure.

Some kind of FieldAccessorFactory registry would be great. I now ended up configuring the whole Spring Data Graph stuff in XML bean notation, because the java config of course does not know anything about my extended NodeDelegatingFieldAccessorFactory and it has to be wired in manually. May there be simpler alternatives to just "replace" the NodeDelegatingFieldAccessorFactory bean with an own implementation?

Interesting to use @RelatedTo for that, I'm a bit leaning towards a custom annotation? (Or probably none).

I've also thought about the solution you're proposing.

What one could do is to have the default-inlined solution for normal Maps and when using @RelatedTo then it is a separate node.

The problem with the inlined map is that this map does not know which properties belong to the map and which belong to the entity itself, when it is loaded. Imagine the following entity:

@NodeEntity
class Person {
    String name;
    PropertyMap properties;
} 

The properties map contains the keys "foo" and "bar". The node then contains the properties "name", "foo" and "bar". When loading the node into the entity, the FieldAccessor for the properties map does not know that the property "name" does not belong to this map, but to the entity itself. I could not quickly find a way to let a FieldAccessor know about the already handled fields. Any ideas?

I'd like to have such a dynamic approach being supported in SDG.

I'm not sure about using a generic Map structure. Probably just using a neo4j Node or a SDG-MapNode or a simple PropertyContainer would be enough? For a Map you have to support all the operations of the Map interface some of which don't fit so well in this context (and it would have to be a managed Map anyway whose content must be reflected in the graph on any change)

I introduced an interface PropertyMap that is the same as PropertyContainer, but without the getGraphDatabase() method. An implementation of this PropertyMap uses a Map<String, Object> as a delegate to actually store the key/value pairs and is used in the DynamicPropertiesFieldAccessor that handles properties of type PropertyMap.

My next step will be to try to store the dynamic map in a separate node so that I do not have to know which keys are already handled by other FieldAccessors on the entity itself.


Best regards,
James

[ractive]

Some kind of FieldAccessorFactory registry would be great. I now ended up configuring the whole Spring Data Graph stuff in XML bean notation, because the java config of course does not know anything about my extended NodeDelegatingFieldAccessorFactory and it has to be wired in manually. May there be simpler alternatives to just "replace" the NodeDelegatingFieldAccessorFactory bean with an own implementation?

Ohh... Extending the Neo4jConfiguration and overriding the nodeEntityStateFactory() method is enough to just re-define this bean.

@Configuration
public class ExtendedNeo4jConfiguration extends Neo4jConfiguration {
    @Override
    @Bean
    public NodeEntityStateFactory nodeEntityStateFactory() throws Exception {
        final GraphDatabaseContext graphDatabaseContext = graphDatabaseContext();
        final DirectGraphRepositoryFactory graphRepositoryFactory = directGraphRepositoryFactory();

        NodeEntityStateFactory entityStateFactory = new NodeEntityStateFactory();
        entityStateFactory.setGraphDatabaseContext(graphDatabaseContext);
        entityStateFactory.setEntityManagerFactory(getEntityManagerFactory());
        final ExtendedNodeDelegatingFieldAccessorFactory nodeDelegatingFieldAccessorFactory = new ExtendedNodeDelegatingFieldAccessorFactory(
                graphDatabaseContext);
        entityStateFactory.setNodeDelegatingFieldAccessorFactory(nodeDelegatingFieldAccessorFactory);
        return entityStateFactory;
    }
} 

So this actually is enough to wire in my "ExtendedNodeDelegatingFieldAccessorFactory". I never worked with java config before, but this is gives me a reason to have a look at it. This is quite a nice way to override configurations.

[MichaelHunger]

I just hacked together a proof of concept implementation that stores all key/value pairs of a PropertyMap field of a NodeEntity to the underlying neo4j node. PropertyMap is an newly introduced PropertyContainer like interface (see below). A own FieldAccessorFactory handles getting and setting this field.

Sounds good.

Some kind of FieldAccessorFactory registry would be great. I now ended up configuring the whole Spring Data Graph stuff in XML bean notation, because the java config of course does not know anything about my extended NodeDelegatingFieldAccessorFactory and it has to be wired in manually. May there be simpler alternatives to just "replace" the NodeDelegatingFieldAccessorFactory bean with an own implementation?

Right. You shouldn't need to configure everything in XML just override the definition of "nodeEntityStateFactory" with an equivalent of this bean definition:

    @Bean
    public NodeEntityStateFactory nodeEntityStateFactory() throws Exception {
        NodeEntityStateFactory entityStateFactory = new NodeEntityStateFactory();
        entityStateFactory.setGraphDatabaseContext(graphDatabaseContext());
        entityStateFactory.setEntityManagerFactory(entityManagerFactory);
        final NodeDelegatingFieldAccessorFactory nodeDelegatingFieldAccessorFactory = new NodeDelegatingFieldAccessorFactory(graphDatabaseContext());
        entityStateFactory.setNodeDelegatingFieldAccessorFactory(nodeDelegatingFieldAccessorFactory);
        return entityStateFactory;
    } 

i.e. something like this should be enough

   <bean id="nodeEntityStateFactory" class="NodeEntityStateFactory">
      <property name="graphDatabaseContext" ref="graphDatabaseContext"/>
      <property name="entityManagerFactory" ref="entityManagerFactory"/>
      <property name="nodeDelegatingFieldAccessorFactory">
          <bean class="MyNodeDelegatingFieldAccessorFactory">
               <constructor-arg ref="graphDatabaseContext"/>
          </bean>
      </property>
   </bean> 

The problem with the inlined map is that this map does not know which properties belong to the map and which belong to the entity itself, when it is loaded.

I'd solve that by prefixing the properties with "fieldName." just as you proposed in your initial post. So you can easily filter the properties that belong to your map by iterating over node.getPropertyNames() and filtering for the field.getName() that you can get from your FieldAccessorFactory.

I introduced an interface PropertyMap that is the same as PropertyContainer, but without the getGraphDatabase() method. An implementation of this PropertyMap uses a Map<String, Object> as a delegate to actually store the key/value pairs and is used in the DynamicPropertiesFieldAccessor that handles properties of type PropertyMap.

My next step will be to try to store the dynamic map in a separate node so that I do not have to know which keys are already handled by other FieldAccessors on the entity itself.

Looking forward to that and to merge it into SDG itself.

Thanks a lot for your efforts.

Michael

[ractive]

I'm on vacation for the next two weeks. I did not manage to prepare this stuff to be merged into SDG before my holiday. Storing of inline properties works so far. The next step will be to store properties on a separate node. I had no time to work on this yet.
So expect it to be ready in around three weeks.