BioText UK

biotext.org.uk launched…

andrewclegg — Fri, 19 Dec 2008 11:11:37 +0000

I’ve launched this site’s successor, at biotext.org.uk.

Please update your links — all new material will be posted there instead.

Andrew.

Querying MeSH with XQuery and SQL

andrewclegg — Tue, 06 May 2008 15:11:40 +0000

This is the third in a series of posts on MeSH headings and how they can be manipulated in XML form in XML databases like SQL Server. Before reading this, you’ll want to read the previous post on loading the data, which in turn refers to the original post where I described the schema.

Most of the queries shown below use a combination of XQuery and Transact-SQL (Microsoft’s dialect). Some of them might well be achievable in pure XQuery, but in practice I found that it was a lot easier to express my intention using hybrid queries.

A similar compromise exists in the storage format. I originally loaded the XML data into a one-column, one-row table — that is, the entire MeSH tree in a single cell. However, many queries turned out to be rather awkward like this, so I chopped up the tree into individual descriptor records, and held each such record in its own row in the table. I also ‘hoisted’ the unique identifier for each descriptor out of the XML and into a separate char column, for ease of access. If I was working on a system that relied on repeated execution of similar queries I’d probably ‘hoist’ more elements out of the XML and into their own columns (or even lookup tables), for performance and ease of access. This is a good compromise between a pure XML/XQuery solution, and converting your XML schema into an entirely relational model.

The queries

First up, a few checks to make sure everything looks happy:

--  Display the schema, reconstructed from SQL Server's internal representation

select xml_schema_namespace( 'MeSH', 'DescriptorSchema' );

-- Count number of descriptors to check we've loaded everything
-- correctly: should be 24767 (April 2008 MeSH)

select count( * ) from [MeSH].[Descriptor];

-- Select a single descriptor as an XML fragment

select top 1 * from [MeSH].[Descriptor];

select * from [MeSH].[Descriptor] where ui = 'D009125';

There’s essentially two ways to retrieve XML data, as XML fragments or cast into native SQL data types:

-- Get all tree numbers (as XML lists)

with xmlnamespaces( default 'uri:mesh' )
select ui, record.query( '
DescriptorRecord/TreeNumberList/TreeNumber
' ) as TreeNumber
from [MeSH].[Descriptor];


-- Get all tree numbers (flattened out)

with xmlnamespaces( default 'uri:mesh' )
select ui, c.value( '.', 'char( 7 )' ) as TreeNumber
from [MeSH].[Descriptor]
cross apply record.nodes( '
DescriptorRecord/TreeNumberList/TreeNumber
' ) as T( c );

The last two queries demonstrated pretty simple XQuery fragments embedded in SQL (the DescriptorRecord/TreeNumberList/TreeNumber bits which define the elements of interest). Of course, these can be as complex as you like. The next one shows how to look for all String elements that occur in the names of descriptors, concepts or terms — or more specifically, which aren’t in the context of qualifiers or related terms (which I’m not covering yet):

-- Get all descriptor/concept/term strings (distinct per descriptor)

with xmlnamespaces( default 'uri:mesh' )
select distinct ui, c.value( '.', 'varchar( 4000 )' ) as String
from [MeSH].[Descriptor]
cross apply record.nodes( '
DescriptorRecord//String
	[ local-name( .. ) != "QualifierName"
		and local-name( ../.. ) != "DescriptorReferredTo" ]
' ) as T( c );

One useful starting point for these queries is to find all descriptors who match a given substring (at the level of descriptor name, concept names or terms). However, somehow Microsoft managed to make the XQuery contains function case-sensitive, without implementing functions to force lower or upper case, so to do case-insensitive string comparisons you have to do them with likeoutside of the XQuery:

-- Get all descriptors matching a query string (at D, C or T level)

declare @query varchar( 50 );
set @query = '%muscle%';
with xmlnamespaces( default 'uri:mesh' ),
T1 as
(
	select ui, c.value( '.', 'varchar( 4000 )' ) as String
	from [MeSH].[Descriptor]
	cross apply record.nodes( '
	DescriptorRecord//String
		[ local-name( .. ) != "QualifierName"
			and local-name( ../.. ) != "DescriptorReferredTo" ]
	' ) as T( c )
)
select ui from T1
where String like @query;

So that’s one of the hierarchies within MeSH, the descriptor-concept-term hierarchy. Traversing the other hierarchy, the tree of TreeNumbers, is easy to achieve using substring comparisons, either in XQuery or SQL. Putting these together, you can extract data from all the descriptors, concepts and terms within a particular branch (or branches) of the tree:

-- Get all strings at or under a given tree number (remove % from
-- end of @query to restrict to exact node without descendents)

declare @query varchar( 50 );
set @query = 'A02.633%';
with xmlnamespaces( default 'uri:mesh' ),
T1 as
(
	select ui, c.value( '.', 'varchar( 50 )' ) as TreeNumber
	from [MeSH].[Descriptor]
	cross apply record.nodes( '
	DescriptorRecord/TreeNumberList/TreeNumber
	' ) as T( c )
),
T2 as
(
	select ui, c.value( '.', 'varchar( 4000 )' ) as String
	from [MeSH].[Descriptor]
	cross apply record.nodes( '
	DescriptorRecord//String
		[ local-name( .. ) != "QualifierName"
			and local-name( ../.. ) != "DescriptorReferredTo" ]
	' ) as T( c )
)
select T2.ui, T2.String
from T2 inner join T1 on T2.ui = T1.ui
where T1.TreeNumber like @query;

Finally, the query that I needed in the first place, which inspired all this hacking. Given an input string, I wanted to get all the strings in subtrees headed by descriptors matching the input string. This involves first identifying all the descriptors that match the input string, at any descriptor/concept/term level, retrieving all of their TreeNumbers, and then extracting the strings associated with these descriptors and any other descriptors that are descendents of these in the TreeNumber hierarchy.

This requires one expression (T1) that models the TreeNumber hierarchy, and another (T2) to model the hierarchy internal to each descriptor, just like the previous query. However, in this case we need two copies of each of these in the SQL join that brings it all together. This is because the query maps from input string -> TreeNumbers -> descendent TreeNumbers -> output strings. So it’s a little more complicated than before:

-- Get all strings at or under tree nodes matching a given query string
-- (remove %s from @query to match exact string only, no substring matches)

declare @query varchar( 100 );
set @query = '%Muscle%';
with xmlnamespaces( default 'uri:mesh' ),
T1 as
(
	select ui, c.value( '.', 'varchar( 50 )' ) as TreeNumber
	from [MeSH].[Descriptor]
	cross apply record.nodes( '
	DescriptorRecord/TreeNumberList/TreeNumber
	' ) as T( c )
),
T2 as
(
	select ui, c.value( '.', 'varchar( 4000 )' ) as String
	from [MeSH].[Descriptor]
	cross apply record.nodes( '
	DescriptorRecord//String
		[ local-name( .. ) != "QualifierName"
			and local-name( ../.. ) != "DescriptorReferredTo" ]
	' ) as T( c )
)
select distinct T2_out.ui, T2_out.String
from T2 T2_out inner join T1 T1_out on T2_out.ui = T1_out.ui
	inner join T1 T1_in on T1_out.TreeNumber like T1_in.TreeNumber + '%'
	inner join T2 T2_in on T2_in.ui = T1_in.ui
where T2_in.String like @query
order by String asc;

I’m not going to give a line-by-line breakdown of how this works, but if you familiarize yourself with the SQL Server XQuery docs it should become clear. Leave a comment if you get stuck though. Have fun!

Andrew.

Loading MeSH into SQL Server 2005

andrewclegg — Thu, 24 Apr 2008 16:47:48 +0000

One of the neat things you can do with many modern relational database servers is import and index hierarchically-structured XML data in a similar way as you would with tabular data — without ‘shredding’ it into tables and rows.

Of course, you can also do this with native XML databases or other query engines, but the advantage of using an RDBMS is that you can integrate queries over XML documents with regular SQL queries.

I was writing some custom, task-specific Perl tools for querying the MeSH controlled vocabulary when it occurred to me that a more flexible, general-purpose solution might be possible by loading it all into SQL Server which we use for normal database purposes anyway (just the free version behind that link). It worked pretty well and I thought I’d share my methods in case anyone else would find it useful.

The first thing you need is the MeSH descriptors XML distribution, available here (the link entitled ’2008 MeSH in XML format’).

Then you import the XSD schema for MeSH into SQL Server. Well, you don’t actually need to, but it will allow SQL Server to validate the XML as it loads (always a good thing) and is supposed to make things faster too. To do this, you need a little bit of T-SQL as follows:

create xml schema collection [MeSH].[DescriptorSchema] as
'
paste_the_XSD_schema_in_here
'

Remember those single-quotes!

Then you can create, load and index the table to hold the XML. It has two columns, one to hold the unique identifier of each descriptor record, and the other to hold the entire descriptor record in parsed XML format.

First, create the table.

create table [MeSH].[Descriptor]
(
   ui char( 7 ) PRIMARY KEY,
   record XML( content [MeSH].[DescriptorSchema] )
);

Then create a temporary table to load the entire MeSH file into, for shredding.

create table [MeSH].[DescriptorTemp]
(
	data xml( document [MeSH].[DescriptorSchema] )
);

Load the data into this table to start with. Obviously, replace m:\desc2008.xml with the location you saved the MeSH XML distribution to.

insert into [MeSH].[DescriptorTemp]
	select *
		from openrowset( bulk 'm:\desc2008.xml', single_blob ) as data;

This is where it starts to get complex. For each descriptor record, pull out the unique identifier, and store the record with the identifier in two columns of a new row in the main table. (CROSS APPLY is very handy for shredding XML.)

with xmlnamespaces( default 'uri:mesh' )
insert into [MeSH].[Descriptor] select
	T.c.value( './DescriptorUI', 'char( 7 )' ) as ui,
	T.c.query( '.' ) as record
from [MeSH].[DescriptorTemp]
cross apply data.nodes( '/DescriptorRecordSet/DescriptorRecord' ) as T( c );

Then we need to index the XML data column so it’s queriable.

create primary xml index DescriptorIdx on [MeSH].[Descriptor]( record );

Finally, clean up.

drop table [MeSH].[DescriptorTemp];

Now it’s ready to use, and I’ll give some handy example queries in the next post.

In theory, adding additional indices to the XML column should improve performance, but I haven’t found a combination of indices yet that really makes my queries much faster. I suspect it depends very much on how your queries work, and in particular how much happens in the XML as opposed to relational parts of the query.

Andrew.

XSD schema for MeSH

andrewclegg — Mon, 21 Apr 2008 16:39:45 +0000

The National Library of Medicine distributes the entire 25,000-node tree of MeSH descriptors (Medical Subject Headings) in XML form. Although they supply a DTD, many XML tools these days are designed to use XML schemas (aka XSDs) only.

I’ve converted the MeSH DTD into an XSD, starting with an automatic conversion using the W3C’s dtd2xsd.pl script, and then manually correcting the output. I haven’t done much XSD for a while so I’d be grateful for any feedback on it. I’ve included an explanation of the structure of MeSH because it has some subtleties that can make querying it rather confusing.

A note on MeSH’s structure

One thing I didn’t entirely realise about MeSH before I embarked on this project was that it’s actually hierarchical along two entirely orthogonal axes. Say what now? Put another way, there’s a hierarchical structure both between and within the descriptors.

Each descriptor (D) is composed of one or more related concepts (C), which can each in turn be referred to by one or more terms (T). For example, consider descriptor D27.505.954.122.085, Anti-Bacterial Agents:

D Anti-Bacterial Agents
  C Anti-Bacterial Agents
    T Anti-Bacterial Agents
    T Antibacterial Agents
  C Anti-Mycobacterial Agents
    T Anti-Mycobacterial Agents
    T Antimycobacterial Agents
  C Antibiotics
    T Antibiotics
  C Bacteriocidal Agents
    T Bacteriocidal Agents
    T Bacteriocides

In general then, the concepts account for subtle differences in meaning that are too fine to use as the basis for a whole new descriptor, and the terms account for different ways of describing the same concept (e.g. spelling or phrasing variations). This hierarchy is represented explicitly within the XML schema.

However, the descriptors themselves are organized into a hierarchical structure of (apparently) unrestricted depth, and these relationships are only implicitly shown in the XML via the ‘tree numbers’ of the descriptors themselves. For example, the parent of Anti-Bacterial Agents [D27.505.954.122.085] is Anti-Infective Agents [D27.505.954.122], and its children include Antitubercular Agents [D27.505.954.122.085.255] and Leprostatic Agents [D27.505.954.122.085.777]. All of these ultimately descend from Chemical Actions and Uses [D27]. It’s all in the numbers, see — a bit like CATH codes and EC numbers.

But wait, there’s one more level of complexity. Viewed from the point of view of the tree numbers, MeSH is a forest of proper trees, but each descriptor can have more than one tree number. This means that from the perspective of the descriptors themselves, MeSH isn’t a tree but a directed graph of parent-child relations. It’s not a tree because a descriptor with multiple tree numbers has multiple parents. And it’s cyclic, because you can follow parent-to-child paths and end up back at the descriptor you started at. This results in some surprising circuits. For example, Morals [F01.829.500] is the parent of Ethics [F01.829.500.519] on the Behavior and Behavior Mechanisms [F01] tree, but Morals [K01.316.630] is a child of Ethics [K01.316] on the Humanities [K01] tree. Watch out for infinite loops if you write any code to traverse MeSH by descriptor.

Note that while the descriptor-concept-term hierarchy is enforced by the schema, the tree number hierarchy can’t be, although it is pretty easy to check using substring matching. Note also that there are other elements in MeSH that don’t fit into this structure, such as pharmacological actions, related terms and subheadings, but let’s save at least some material for a later post.

Andrew.

The Schema

Hello world

andrewclegg — Fri, 18 Apr 2008 10:22:25 +0000

Welcome to BioText UK.