Currently the VLDockingViewDescriptor does not support DockGroups so there
is no way to enforce them through configuration as far as I have found short of extending VLDockingApplicationPage, VLDockingAplicationPageFactory, VLDockingViewDescriptor, and ViewDescriptorDockable and adding DockGroup support.
I have created an issue in jira for this and submitted a patch that adds this functionality. If you need this support, please vote on it. I'm just waiting for it to be imported into the codebase.
http://jira.springsource.org/browse/RCP-609
The only way I know how to open a view as a tab is to extend VLDockingApplicationPage and override the doAddPageComponent(..) method or to provide the VLDockingPageDescriptor with your custom implementation of VLDockingLayoutManager. The default VLDockingLayoutManager simply calls desktop.addDockable(dockable) and desktop.remove(dockable) to add and remove dockable views. I would say the second method is probably preferred.
Although I haven't tried it i suppose you could somehow embed DockGroup support in your layout manager code as a side effect of adding/removing views.