KFDI (kernel file system - database interface)
& EXS File System
﹛
1.1 What is KFDI
KFDI is an interface between file systems and databases in the Linux kernel. With the help of KFDI, file management and searching can be based on data properties. KFDI also introduces some object-oriented concepts into file organization.
KFDI is under the GNU GPL (see file "COPYING"). KFDI has not been strictly tested. It comes with NO WARRANTY.
The aim of KFDI is to build a bridge between conventional hierarchy file systems and databases, so that these two most important storage technologies can cooperate in data management. Taking advantage of relational database technology and XML, KFDI enables users to manage their data in a more flexible and efficient way.
EXS is an example file system that supports KFDI. The name "EXS" implies three involved technologies: ext2 file system, SQL database, and XML. EXS is derived from the ext2 file system. Actually, the modification is very small. It's distribution should follow the same license as the original ext2 source code.
Files are now separated into classes. Each file class has a set of attributes and an ID which is unique in a file volume. Inheritance is supported. Actually any file class except the root class (fclass 0) is derived from a single parent class. The root file class, which is the root of inheritance tree, comprises basic file attributes. All other file classes inherit these attributes. Refer to XML schema file "root.xsd" for the definition of the root fclass.
The user can register a file to KFDI and assign a file class to it; thus an instance of that file class is created, called a file object. For each file volume that supports KFDI, there is a corresponding database named after that file volume. File classes correspond to tables named like Ti (i is file class ID) in the database; the attribute members of fclass i are mapped to the columns of table Ti; each file object of fclass i has an entry in table Ti. When a file volume is mounted, a kernel thread called kfda (kernel FS - DB agent) is started. It is responsible for maintaining the file objects in the volume.
The root attributes (attributes belong to root fclass) of a file object can only be updated inside the kernel according to its basic FS attributes. The user can choose to let kfda automatically maintain the consistency of root attributes between the database and file system, or manually request kfda to update database entries when needed.
KFDI supports displaying a file object in XML format and describing file classes with XML schema. The user can input XML description to update the DB track of a file (root attributes can not be updated in this way), or use XML schema to define a new file class. Currently, only eight simple XML schema types are supported. They are: xs:byte, xs:short, xs:int/integer, xs:float, xs:double, xs:boolean, xs:char, and xs:string. A valid XML schema may look like below:
1 <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
2 <xs:element name="photo"/>
3 <xs:annotation>
4 <xs:documentation>this is the file class for my photo files</xs:documentation>
5 </xs:annotation>
6 <xs:complexType name="photoType" id="100">
7 <xs:complexContent>
8 <xs:extension base="pictureType" id="50">
9 <xs:all>
10 <xs:element name="date">
11 <xs:simpleType>
12 <xs:restriction base="xs:string">
13 <xs:maxLength value="255"/>
14 </xs:restriction>
15 </xs:simpleType>
16 </xs:element>
17 <xs:element name="site" type="xs:string">
18 <xs:element name="is_there_me" type="xs:boolean"/>
19 </xs:all>
20 </xs:extension>
21 </xs:complexContent>
22 </xs:complexType>
23 </xs:schema>
This example defines a file class named photo (line 2), whose ID is 100 (line 6). File class picture (line 8) whose ID equals 50 is the parent class of photo. More example XML and XML schema files can be found in directory "xml".
KFDI is an open framework over the VFS layer of Linux kernel. It provides a FS interface. Any file system can communicate with databases through this interface, and KFDI is transparent to those file systems that not support it. In fact, many existing file systems can be easily modified to support KFDI.
For databases, the situation is similar. Though currently backed primarily by PostgreSql-7.4.x, it is possible to import new DB severs into the KFDI system, and even a central database in network environment can be used. At mounting, a file system can freely select which DB server to use. To make a database server support KFDI, a simple driver inside the kernel for that database is needed, and the driver must realize the KFDI-DB interface. Although care has been taken to accord with SQL99, KFDI requires DB servers to support inheritance. And currently KFDI adopts PostgreSql＊s syntax to express the inheritance relationship, which means if a DB server can not understand this syntax, its kernel driver may have to do some translations.
KFDI preserves the fclass ID and other DB relevant information of a file object in the Linux VFS dentry structure. The common information of a mounted volume is preserved in the root dentry.
KFDI exports four functions to help file systems request database services.
1) void kfdi_setup_dbinfo (struct super_block *psb, Byte db_id, char *fs_name)
kfdi_setup_dbinfo should be called when mounting a file volume. It setups the database information of the volume, and starts a kfda thread. Parameter db_id specifies the DB server the file system wants to connect; currently only value 1 (for PostgreSql) is valid.
2) void kill_kfda (struct super_block *psb)
kill_kfda is called when unmounting a file volume. Its main task is to wait the kfda thread exit and release relevant resources.
3) void request_kfda (Byte type, Byte fclass, uInt ino, struct dentry *pdentry)
request_kfda is used to request kfda to update the DB entry of a file object. Parameter type indicates the type of a request, which could be: KFDA_UPDATE_ATTR (invoked when root attributes alter), KFDA_RENAME (file is renamed or moved), KFDA_DEL_ENTRY (file is removed).
4) int kfdi_ioctl (uInt cmd, struct dentry *pdentry, uLong arg)
kfdi_ioctl is called from the FS ioctl function. It is used to get/set database related properties of a file object, or to register/unregister a file object. Parameter cmd and arg are passed from system call sys_ioctl. For details, please refer to section 3.3 每 User Interface.
The KFDI-DB interface consists of
four function prototypes.struct kfdi_db_operations {
void (*connect_db) (struct db_con *pcon);
void (*exec_query) (struct db_con *pcon);
void (*close_db) (struct db_con *pcon);
/* describe a file class with XML schema */
void (*fclass_schema) (struct db_con *pcon);
}
In all these functions, parameter pcon indicates a DB connection. All connection status and the input/output of DB operations are preserved in this structure.
For more details, see the annotation in source file "kfdi_internal.h".
Users or user-space applications can interact with KFDI through two ways.
1) system call sys_ioctl
It will lead to the invocation of kfdi_ioctl. KFDI defines five new ioctl commands. They are:
﹞ KFDI_IOC_ADDDBTRACK 每 Register/unregister a file object. Requires write permission on the target file.
﹞ KFDI_IOC_GETFCLASSID 每 Get the file class ID of a file object.
﹞ KFDI_IOC_READDBTRACK 每 Output the XML description of a file object.
﹞ KFDI_IOC_UPDATEDBTRACK 每 Update the attributes of a file object in the database. Accepts XML format inputs. Requires write permission and cannot operate on a read-only file.
﹞ KFDI_IOC_AUTODBTRACK 每 Specify whether the attributes of a file object is automatically maintained.
For more details see the annotation in source file "ioctl.c".
2) system call: int sys_kfdi_call (Byte cmd, char *fs_info, int *pinlen, int *poutlen, char *inbuf, char *outbuf)
This is a new system call added by KFDI. Its function depends on the value of argument cmd, which could be:
﹞ KFDI_SYS_NEWFCLASS 每 Add a new file class to a file volume. Accepts XML schema format inputs. Requires Linux CAP_ADMIN capability.
﹞ KFDI_SYS_DELFCLASS 每 Delete a file class. Requires the CAP_ADMIN capability.
﹞ KFDI_SYS_DESCRIPFCLASS 每 Output the XML schema description of a file class.
﹞ KFDI_SYS_MODIFYFCLASS 每 Rename a file class or modifies a file class definition. Requires the CAP_ADMIN capability.
﹞ KFDI_SYS_COMMONQUERY 每 Perform SQL update and select query. The input SQL sentences will be checked by KFDI.
﹞ KFDI_SYS_ADMINQUERY 每 Execute any server supported DB commands. Requires the CAP_ADMIN capability.
The difference between COMMONQUERY and ADMINQUERY is that: the inputs (SQL sentences) to the former operation must pass the security check by KFDI. For details about the checking, refer to section 4 每 KFDI Security.
For more details see the annotation in source file "syscall.c".
3.4 EXS File System
In the EXS file system implementation, the file volume name is used as the name of corresponding database; the DB server ID is preserved in structure ext2_super_block; the fclass ID of file objects is preserved in the file_type field of structure ext2_dir_entry_2.
Though EXS is compatible with ext2 file system in the kernel, once an EXS file volume is mounted as ext2 type, the file classes of file objects may be incorrectly altered.
KFDI takes efforts to ensure the security of information in databases, however to a large extent, the security depends on the proper configuration of DB servers. As for PostgreSql, we should configure that only the system administrator can connect to a KFDI database under PostgreSql identity postgres, and only PostgreSql super user postgres can directly access the database.
KFDI will check the SQL query sentences inputted by an ordinary user, but privileged users (holding CAP_ADMIN) can perform any server supported commands through ADMINQUERY. A rough description of the checking is shown below. For details see source file "check.c".
Condition expressions that enforce access control will be appended to the where clause of inputted SQL sentences. The access control over the database entries of file objects is based on files' ACL. Only when a process holds write permission of a file, can it update the file object attributes in the database; and only the processes that hold read permission of the file can query the file object attributes. However, processes with CAP_OVERRIDE_DAC capability can override this rule. Currently parent directories＊ ACL is not taken into account.
The target query table can only be Ti (i is the fclass ID); updating root attributes is not allowed; semicolons can not appear in a query sentence; the checking also ensure that left and right parentheses, quotations, and single quotations appear in pairs.
Currently only select and update query are supported by COMMONQUERY. Nested queries in any form are not permitted; select into, for update (and still others) are not supported; it is possible to use union to combine the results of several queries; transaction is only supported by ADMINQUERY.
There are two ways to build KFDI and EXS kernel modules:
(1) Direct building
i. Specify BINDIR (installation path) and LINUXDIR (Linux kernel source tree) in "mk/base.mk";
ii. "make; make install"
(2) Building through kernel patch
i. Change into Linux kernel source directory;
ii. Apply the patch by "patch -p1 < linux-2.4.x-kfdi-x.patch";
iii. Config and build the kernel.
To support KFDI, the kernel must have sys_call_table exported, for KFDI will register a new system call through it. Both KFDI and EXS can be statically linked into the kernel or built as a kernel module.
In directory "utils", there are several little applications that can help you experience KFDI. They are kfdi_pg_createdb, kfdi_ioctl, and kfdi_call. To build them, just change into the directory and "make".
kfdi_pg_createdb is a simple shell script used to create a new database on the local PostgreSql server. Prior to the first using, you should specify the "bin" directory path of your PostgreSql installation at the beginning of the script file, or you may like to add the path to environment variable $PATH.
You can install PostgreSql-7.4.x through any way, but must use the default administrator name (postgres) and enable the UNIX-domain socket connection with the default path: "/tmp/.s.PGSQL.5432".
i. Format an EXS file volume by "mkfs -t ext2 -L volume_name device", where volume_name is the name (also ext2 volume label) of the new volume.
ii. Create database db_name and two initial tables: Tfclass and T0, where db_name should equal volume_name. If PostgreSql (the only choice currently) is selected, you can use ※~/kfdi_pg_createdb db_name§ to fulfill this job.
In order to have KFDI work properly with PostgreSql, Linux root user must be able to connect to the database under identity postgres via the UNIX-domain socket.
Use "mount -t exs device path -o options" to mount a EXS volume, while the DB server must be running. EXS adds three new mount options:
﹞ "db_server=id" 每 Used to specify the DB server ID. At the first mounting of a volume, this option must be provided.
﹞ "fs_name=name" 每 Specify the volume (and database) name. This option must be provided if the volume label have not been named when creating the EXS volume.
﹞ without_db 每 Mount a volume without connecting to the KFDI database.
kfdi_ioctl is a simple program that shows how to utilize KFDI through system call sys_ioctl.
Usage:
kfdi_ioctl -a fclass file
kfdi_ioctl -d|g|n|t file
kfdi_ioctl -r|u [-f xmlfile] file
-a: register file object to KFDI
-d: unregister file object
-g: get file class ID
-r: read file object attributes
-u: update file object attributes
-n: not automatically maintain file object
-t : automatically maintain file object
fclass: 8-bits integer. 7 low bits for file class ID; the highest bit is a flag bit denoting whether the file object is maintained by KFDI.
file: target file
xmlfile: input/output XML file
Examples:
1) "kfdi_ioctl -a 129 file"
Register file to KFDI. Set its fclass ID = 1; kfda will not automatically maintain the file object attributes (129 = (1 << 7) + 1).
2) "kfdi_ioctl -u file"
Request KFDI to update only the root attributes of file in database.
3) "kfdi_ioctl -u -f xmlfile file"
Use xmlfile to update the attributes of file in database. xmlfile must not contain root attributes.
kfdi_call is a simple program that shows how to utilize KFDI through system call sys_kfdi_call.
Usage:
kfdi_call -n -i file [-b db] fs
kfdi_call -d fclass [-b db] fs
kfdi_call -p fclass [-o file] [-b db] fs
kfdi_call -m fclass [--name name] [--desc desc] [-b db] fs
kfdi_call -q|a -i file|-s query [-o|f file] [-b db] fs
-n: new file class
-d: delete file class
-p: describe file class with XML schema
-m: modify file class name and description
-q: common database query
-a: administrative database query
-i: use file as input
-o: save result to a file
-f: use file as system call parameter (result will be written to file by KFDI inside kernel)
file: input/output file
fclass: file class ID (0 - 126)
query: SQL query sentence
name: file class name
desc: file class description
db: database server (currently only 'postgresql' is supported)
fs: file volume name
Examples:
1) "kfdi_call -n -i schema -b postgresql volname"
Add a new file class defined in schema to file volume volname, DB server postgresql specified.
2) "kfdi_call -m 100 --desc "fclass for my private photos" volname"
Modify the description of fclass 100.
3) "kfdi_call -q -s "SELECT author,years,fpath FROM ONLY T101 UNION SELECT hero,date,fpath FROM T102 WHERE author LIKE '%Bronte'" -o outfile volname"
Do ordinary DB queries and save the results to outfile.
4) "kfdi_call -q -i sqlfile -f outfile volname"
Do ordinary DB queries. Use sqlfile as input; outfile will be passed to sys_kfdi_call as the output stream.
5) "kfdi_call -a -s "BEGIN;CREATE INDEX ON Tfclass USING HASH (name);ANALYZE;UPDATE ONLY T99 SET author='Bronte sisters' WHERE author LIKE '%Bronte';COMMIT;SELECT COUNT(*) FROM T99" volname"
Execute administrative DB commands and display the results (if any) on terminal.
I wrote KFDI with intension to prove that the cooperation between file system and relational database can be realized in the Linux kernel, and can make an notable improvement on storage management. I wish that one day KFDI or the idea of KFDI is merged into the VFS layer of Linux kernel, and is recognized by more people.
Presently, KFDI is just an exploring prototype. Actually it is rather rough in many aspects. For stability, performance, and facility, there are still lots of exploration to do. If you think KFDI is a good idea, you are welcome to join in the promotion of this idea.
﹛
Any suggestion or comment is welcome.