diff options
15 files changed, 216 insertions, 210 deletions
@@ -1,8 +1,21 @@ -ETDB +METRIC ==== To enable edge computing in its full capacity, a crucial requirement is to manage the state of edge applications, preferably in database that provides the full features of SQL including joins and transactions. The key challenge here is to provide a replicated database for the edge that can scale to thousands of geo-distributed nodes. Existing solutions either provide semantics that are too weak (PostgreSQL replicates asynchronously) or too strong and hence expensive to realize in a geo-distributed network with its WAN latencies and complex failure modes (MariaDb, Spanner, provide full transactionality). Inspired by entry consistency in shared memory systems, wherein only the lock holder for an object obtains sequential consistency for the object, we define the notion of an entry transactional database, which is a novel partitioned database in which only the “owner” of a partition obtains full ACID transactionality. In this work, we define the semantics of an entry transactional database, describe the hard challenges faced in building it and present a novel middleware called mdbc that combines existing SQL databases with an underlying geo-distributed entry consistent store to provide entry transactional semantics. Further, we present crucial use cases such as a federated regional controller for the network control plane and a state management service for edge mobility enabled by entry transactionality. +## Dependencies + +Given that METRIC is still a project in development, we are using newer versions (aka not release) of MUSIC, such that once +it reaches production it matches the latest MUSIC version. So for now we cannot use maven repositories to pull MUSIC. Follow +the next steps to compile and add the right version of music locally: +```bash +cd /tmp/ +git clone https://gerrit.onap.org/r/music +cd music +git checkout dev-cassandra-only +mvn install -Dfile=target/MUSIC.jar -DpomFile=./pom.xml -DskipTests +``` + ## Running METRIC run cassandra (tested with v3.11) @@ -64,25 +77,22 @@ It requires three parameters: src/main/java/com/att/research/mdbc/examples +## Building METRIC - - -## Building ETDB - -ETDB is built with Maven. This directory contains two pom.xml files. +METRIC is built with Maven. This directory contains two pom.xml files. The first (*pom.xml*) will build a jar file to be used by applications wishing to use the -ETDB JDBC driver. +MDBC JDBC driver. The second (*pom-h2server.xml*) is used to built the special code that needs to be loaded -into an H2 server, when running ETDB against a copy of H2 running as a server. +into an H2 server, when running METRIC against a copy of H2 running as a server. -### Building the JBoss ETDB Module +### Building the JBoss METRIC Module There is a shell script (located in `src/main/shell/mk_jboss_module`) which, when run in the mdbc source directory, will create a tar file `target/mdbc-jboss-module.tar` which can be used as a JBoss module. This tar file should be installed by un-taring it in the $JBOSS_DIR/modules directory on the JBoss server. -## Using ETDB +## Using METRIC This package provides a JDBC driver that can be used to mirror the contents of a database to and from Cassandra. The mirroring occurs as a side effect of execute() statements against @@ -96,7 +106,7 @@ communicate the existence of dirty rows to other Cassandra replicas (with the Ca Mixin, the table is called DIRTY\_\_\_\_ and there is only one table). Dirty rows will be copied, as needed back into the database from Cassandra before any SELECT. -### To use directly with JDBC + 1. Add this jar, and all dependent jars to your CLASSPATH. 2. Rewrite your JDBC URLs from jdbc:_yourdb_:... to jdbc:mdbc:.... @@ -106,7 +116,7 @@ Dirty rows will be copied, as needed back into the database from Cassandra befor Class.forName("com.att.research.mdbc.ProxyDriver"); The following properties can be passed to the JDBC DriverManager.getConnection(String, Properties) -call to influence how ETDB works. +call to influence how METRIC works. | Property Name | Property Value | Default Value | |--------------------|--------------------------------------------------------------------------------|---------------| @@ -114,7 +124,7 @@ call to influence how ETDB works. | MDBC\_MUSIC\_MIXIN | The mixin name to use to select the MUSIC mixin to use for this connection. | cassandra2 | | myid | The ID of this replica in the collection of replicas sharing the same tables. | 0 | | replicas | A comma-separated list of replica names for the collection of replicas sharing the same tables. | the value of myid | -| music\_keyspace | The keyspace name to use in Cassandra for all tables created by this instance of ETDB. | mdbc | +| music\_keyspace | The keyspace name to use in Cassandra for all tables created by this instance of METRIC. | mdbc | | music\_address | The IP address to use to connect to Cassandra. | localhost | | music\_rfactor | The replication factor to use for the new keyspace that is created. | 2 | | disabled | If set to true the mirroring is completely disabled; this is the equivalent of using the database driver directly. | false | @@ -182,7 +192,7 @@ probably need to make changes to the _connectionProperties_ attribute. ## Databases Supported -Currently, the following databases are supported with ETDB: +Currently, the following databases are supported with METRIC: * H2: The `H2Mixin` mixin is used when H2 is used with an in-memory (`jdbc:h2:mem:...`) or local file based (`jdbc:h2:path_to_file`) database. @@ -195,29 +205,29 @@ or local file based (`jdbc:h2:path_to_file`) database. ## Testing Mixin Combinations -The files under `src/main/java/com/att/research/mdbc/tests` can be used to test various ETDB +The files under `src/main/java/com/att/research/mdbc/tests` can be used to test various METRIC operations with various combinations of Mixins. The tests are controlled via the file `src/main/resources/tests.json`. More details are available in the javadoc for this package. -## Limitations of ETDB +## Limitations of METRIC -* The `java.sql.Statement.executeBatch()` method is not supported by ETDB. +* The `java.sql.Statement.executeBatch()` method is not supported by METRIC. It is not prohibited either; if you use this, your results will be unpredictable (and probably wrong). * When used with a DB server, there is some delay as dirty row information is copied from a table in the database, to the dirty table in Cassandra. This opens a window during which all sorts of mischief may occur. -* ETDB *only* copies the results of SELECTs, INSERTs, DELETEs, and UPDATEs. Other database -operations must be performed outside of the purview of ETDB. In particular, CREATE-ing or +* METRIC *only* copies the results of SELECTs, INSERTs, DELETEs, and UPDATEs. Other database +operations must be performed outside of the purview of METRIC. In particular, CREATE-ing or DROP-ing tables or databases must be done individually on each database instance. * Some of the table definitions may need adjusting depending upon the variables of your use -of ETDB. For example, the MySQL mixin assumes (in its definition of the ETDB_TRANSLOG table) +of METRIC. For example, the MySQL mixin assumes (in its definition of the METRIC_TRANSLOG table) that all table names will be no more than 255 bytes, and that tables rows (expressed in JSON) will be no longer than 512 bytes. If this is not true, you should adjust, edit, and recompile. -* ETDB is limited to only data types that can be easily translated to a Cassandra equivalent; +* METRIC is limited to only data types that can be easily translated to a Cassandra equivalent; e.g. BIGINT, BOOLEAN, BLOB, DOUBLE, INT, TIMESTAMP, VARCHAR * To find the data types that your database is currently using run the following command: @@ -75,11 +75,6 @@ <version>1.1</version> </dependency> <dependency> - <groupId>org.apache.commons</groupId> - <artifactId>commons-lang3</artifactId> - <version>3.1</version> - </dependency> - <dependency> <groupId>com.att.eelf</groupId> <artifactId>eelf-core</artifactId> <version>1.0.0</version> @@ -92,12 +87,6 @@ <version>2.5</version> <scope>provided</scope> </dependency> - <!-- Music jar from onap --> - <dependency> - <groupId>org.onap.music</groupId> - <artifactId>MUSIC</artifactId> - <version>3.2.0</version> - </dependency> <!-- https://mvnrepository.com/artifact/org.apache.calcite.avatica/avatica-server --> <dependency> <groupId>org.apache.calcite.avatica</groupId> @@ -125,6 +114,11 @@ <artifactId>mariadb-java-client</artifactId> <version>1.1.7</version> </dependency> + <dependency> + <groupId>org.onap.music</groupId> + <artifactId>MUSIC</artifactId> + <version>3.2.0</version> + </dependency> </dependencies> <build> diff --git a/src/main/java/com/att/research/mdbc/ArchiveProcess.java b/src/main/java/com/att/research/mdbc/ArchiveProcess.java index f192430..8290d66 100644 --- a/src/main/java/com/att/research/mdbc/ArchiveProcess.java +++ b/src/main/java/com/att/research/mdbc/ArchiveProcess.java @@ -12,10 +12,9 @@ public class ArchiveProcess { //TODO: This is a place holder for taking snapshots and moving data from redo record into actual tables /** - * This method is called whenever there is a DELETE on a local SQL table, and should be called by the underlying databases - * triggering mechanism. It updates the MUSIC/Cassandra tables (both dirty bits and actual data) corresponding to the SQL DELETE. - * Music propagates it to the other replicas. If the local database is in the middle of a transaction, the DELETEs to MUSIC are - * delayed until the transaction is either committed or rolled back. + * This method is called whenever there is a DELETE on the transaction digest and should be called when ownership changes, if required + * It updates the MUSIC/Cassandra tables (both dirty bits and actual data) corresponding to the SQL DELETE. + * Music propagates it to the other replicas. * @param tableName This is the table on which the select is being performed * @param oldRow This is information about the row that is being deleted */ @@ -26,8 +25,8 @@ public class ArchiveProcess { } /** - * This method is called whenever there is an INSERT or UPDATE to a local SQL table, and should be called by the underlying databases - * triggering mechanism. It updates the MUSIC/Cassandra tables (both dirty bits and actual data) corresponding to the SQL write. + * This method is called whenever there is an INSERT or UPDATE to a the transaction digest, and should be called by an + * ownership chance. It updates the MUSIC/Cassandra tables (both dirty bits and actual data) corresponding to the SQL write. * Music propagates it to the other replicas. If the local database is in the middle of a transaction, the updates to MUSIC are * delayed until the transaction is either committed or rolled back. * diff --git a/src/main/java/com/att/research/mdbc/DatabaseOperations.java b/src/main/java/com/att/research/mdbc/DatabaseOperations.java index 750e918..c896b84 100644 --- a/src/main/java/com/att/research/mdbc/DatabaseOperations.java +++ b/src/main/java/com/att/research/mdbc/DatabaseOperations.java @@ -272,69 +272,6 @@ public class DatabaseOperations { return id; } - /** - * This function creates the TransactionInformation table. It contain information related - * to the transactions happening in a given partition. - * * The schema of the table is - * * Id, uiid. - * * Partition, uuid id of the partition - * * LatestApplied, int indicates which values from the redologtable wast the last to be applied to the data tables - * * Applied: boolean, indicates if all the values in this redo log table where already applied to data tables - * * Redo: list of uiids associated to the Redo Records Table - * - */ - public static void CreateTransactionInformationTable( String musicNamespace, - String transactionInformationTableName) throws MDBCServiceException { - String tableName = transactionInformationTableName; - String priKey = "id"; - StringBuilder fields = new StringBuilder(); - fields.append("id uuid, "); - fields.append("partition uuid, "); - fields.append("latestapplied int, "); - fields.append("applied boolean, "); - //TODO: Frozen is only needed for old versions of cassandra, please update correspondingly - fields.append("redo list<frozen<tuple<text,tuple<text,varint>>>> "); - String cql = String.format("CREATE TABLE IF NOT EXISTS %s.%s (%s, PRIMARY KEY (%s));", - musicNamespace, tableName, fields, priKey); - try { - executeMusicWriteQuery(musicNamespace,tableName,cql); - } catch (MDBCServiceException e) { - logger.error("Initialization error: Failure to create transaction information table"); - throw(e); - } - } - - /** - * This function creates the RedoRecords table. It contain information related to each transaction committed - * * LeaseId: id associated with the lease, text - * * LeaseCounter: transaction number under this lease, bigint \TODO this may need to be a varint later - * * TransactionDigest: text that contains all the changes in the transaction - */ - public static void CreateRedoRecordsTable(int redoTableNumber, String musicNamespace, - String redoRecordTableName) throws MDBCServiceException { - String tableName = redoRecordTableName; - if(redoTableNumber >= 0) { - StringBuilder table = new StringBuilder(); - table.append(tableName); - table.append("-"); - table.append(Integer.toString(redoTableNumber)); - tableName=table.toString(); - } - String priKey = "leaseid,leasecounter"; - StringBuilder fields = new StringBuilder(); - fields.append("leaseid text, "); - fields.append("leasecounter varint, "); - fields.append("transactiondigest text ");//notice lack of ',' - String cql = String.format("CREATE TABLE IF NOT EXISTS %s.%s (%s, PRIMARY KEY (%s));", - musicNamespace, tableName, fields, priKey); - try { - executeMusicWriteQuery(musicNamespace,tableName,cql); - } catch (MDBCServiceException e) { - logger.error("Initialization error: Failure to create redo records table"); - throw(e); - } - } - /** * This function creates the Table To Partition table. It contain information related to */ @@ -462,4 +399,67 @@ public class DatabaseOperations { } } } + + + /** + * This function creates the MusicTxDigest table. It contain information related to each transaction committed + * * LeaseId: id associated with the lease, text + * * LeaseCounter: transaction number under this lease, bigint \TODO this may need to be a varint later + * * TransactionDigest: text that contains all the changes in the transaction + */ + public static void CreateMusicTxDigest(int musicTxDigestTableNumber, String musicNamespace, String musicTxDigestTableName) throws MDBCServiceException { + String tableName = musicTxDigestTableName; + if(musicTxDigestTableNumber >= 0) { + StringBuilder table = new StringBuilder(); + table.append(tableName); + table.append("-"); + table.append(Integer.toString(musicTxDigestTableNumber)); + tableName=table.toString(); + } + String priKey = "leaseid,leasecounter"; + StringBuilder fields = new StringBuilder(); + fields.append("leaseid text, "); + fields.append("leasecounter varint, "); + fields.append("transactiondigest text ");//notice lack of ',' + String cql = String.format("CREATE TABLE IF NOT EXISTS %s.%s (%s, PRIMARY KEY (%s));", musicNamespace, tableName, fields, priKey); + try { + executeMusicWriteQuery(musicNamespace,tableName,cql); + } catch (MDBCServiceException e) { + logger.error("Initialization error: Failure to create redo records table"); + throw(e); + } + } + + /** + * This function creates the TransactionInformation table. It contain information related + * to the transactions happening in a given partition. + * * The schema of the table is + * * Id, uiid. + * * Partition, uuid id of the partition + * * LatestApplied, int indicates which values from the redologtable wast the last to be applied to the data tables + * * Applied: boolean, indicates if all the values in this redo log table where already applied to data tables + * * Redo: list of uiids associated to the Redo Records Table + * + */ + public static void CreateMusicRangeInformationTable(String musicNamespace, String musicRangeInformationTableName) throws MDBCServiceException { + String tableName = musicRangeInformationTableName; + String priKey = "id"; + StringBuilder fields = new StringBuilder(); + fields.append("id uuid, "); + fields.append("partition uuid, "); + fields.append("latestapplied int, "); + fields.append("applied boolean, "); + //TODO: Frozen is only needed for old versions of cassandra, please update correspondingly + fields.append("redo list<frozen<tuple<text,tuple<text,varint>>>> "); + String cql = String.format("CREATE TABLE IF NOT EXISTS %s.%s (%s, PRIMARY KEY (%s));", musicNamespace, tableName, fields, priKey); + try { + executeMusicWriteQuery(musicNamespace,tableName,cql); + } catch (MDBCServiceException e) { + logger.error("Initialization error: Failure to create transaction information table"); + throw(e); + } + } + + + } diff --git a/src/main/java/com/att/research/mdbc/DatabasePartition.java b/src/main/java/com/att/research/mdbc/DatabasePartition.java index 6046801..a9b4f3e 100644 --- a/src/main/java/com/att/research/mdbc/DatabasePartition.java +++ b/src/main/java/com/att/research/mdbc/DatabasePartition.java @@ -7,7 +7,6 @@ import java.util.HashSet; import java.util.Set; import com.att.research.logging.EELFLoggerDelegate; -import com.att.research.mdbc.mixins.CassandraMixin; import com.google.gson.Gson; import com.google.gson.GsonBuilder; @@ -19,9 +18,9 @@ import com.google.gson.GsonBuilder; public class DatabasePartition { private transient static EELFLoggerDelegate logger = EELFLoggerDelegate.getLogger(DatabasePartition.class); - private String transactionInformationTable;//Table that currently contains the REDO log for this partition - private String transactionInformationIndex;//Index that can be obtained either from - private String redoRecordsTable; + private String musicRangeInformationTable;//Table that currently contains the REDO log for this partition + private String musicRangeInformationIndex;//Index that can be obtained either from + private String musicTxDigestTable; private String partitionId; private String lockId; protected Set<Range> ranges; @@ -35,7 +34,7 @@ public class DatabasePartition { ranges = new HashSet<>(); } - public DatabasePartition(Set<Range> knownRanges, String titIndex, String titTable, String partitionId, String lockId, String redoRecordsTable) { + public DatabasePartition(Set<Range> knownRanges, String mriIndex, String mriTable, String partitionId, String lockId, String musicTxDigestTable) { if(knownRanges != null) { ranges = knownRanges; } @@ -43,25 +42,25 @@ public class DatabasePartition { ranges = new HashSet<>(); } - if(redoRecordsTable != null) { - this.setRedoRecordsTable(redoRecordsTable); + if(musicTxDigestTable != null) { + this.setMusicTxDigestTable(musicTxDigestTable); } else{ - this.setRedoRecordsTable(""); + this.setMusicTxDigestTable(""); } - if(titIndex != null) { - this.setTransactionInformationIndex(titIndex); + if(mriIndex != null) { + this.setMusicRangeInformationIndex(mriIndex); } else { - this.setTransactionInformationIndex(""); + this.setMusicRangeInformationIndex(""); } - if(titTable != null) { - this.setTransactionInformationTable(titTable); + if(mriTable != null) { + this.setMusicRangeInformationTable(mriTable); } else { - this.setTransactionInformationTable(""); + this.setMusicRangeInformationTable(""); } if(partitionId != null) { @@ -79,20 +78,20 @@ public class DatabasePartition { } } - public String getTransactionInformationTable() { - return transactionInformationTable; + public String getMusicRangeInformationTable() { + return musicRangeInformationTable; } - public void setTransactionInformationTable(String transactionInformationTable) { - this.transactionInformationTable = transactionInformationTable; + public void setMusicRangeInformationTable(String musicRangeInformationTable) { + this.musicRangeInformationTable = musicRangeInformationTable; } - public String getTransactionInformationIndex() { - return transactionInformationIndex; + public String getMusicRangeInformationIndex() { + return musicRangeInformationIndex; } - public void setTransactionInformationIndex(String transactionInformationIndex) { - this.transactionInformationIndex = transactionInformationIndex; + public void setMusicRangeInformationIndex(String musicRangeInformationIndex) { + this.musicRangeInformationIndex = musicRangeInformationIndex; } /** @@ -180,11 +179,11 @@ public class DatabasePartition { this.lockId = lockId; } - public String getRedoRecordsTable() { - return redoRecordsTable; + public String getMusicTxDigestTable() { + return musicTxDigestTable; } - public void setRedoRecordsTable(String redoRecordsTable) { - this.redoRecordsTable = redoRecordsTable; + public void setMusicTxDigestTable(String musicTxDigestTable) { + this.musicTxDigestTable = musicTxDigestTable; } } diff --git a/src/main/java/com/att/research/mdbc/MusicSqlManager.java b/src/main/java/com/att/research/mdbc/MusicSqlManager.java index 1d7bf9e..f73bdb6 100755 --- a/src/main/java/com/att/research/mdbc/MusicSqlManager.java +++ b/src/main/java/com/att/research/mdbc/MusicSqlManager.java @@ -1,12 +1,7 @@ package com.att.research.mdbc; import java.sql.Connection; -import java.util.ArrayList; -import java.util.Collections; -import java.util.HashMap; -import java.util.HashSet; -import java.util.Properties; -import java.util.Set; +import java.util.*; import org.json.JSONObject; @@ -296,4 +291,18 @@ public class MusicSqlManager { */ return musicKeys; } + + public void own(List<Range> ranges) { + throw new java.lang.UnsupportedOperationException("function not implemented yet"); + } + + public void appendRange(String rangeId, List<Range> ranges) { + throw new java.lang.UnsupportedOperationException("function not implemented yet"); + } + + public void relinquish(String ownerId, String rangeId) { + throw new java.lang.UnsupportedOperationException("function not implemented yet"); + } + + } diff --git a/src/main/java/com/att/research/mdbc/StateManager.java b/src/main/java/com/att/research/mdbc/StateManager.java index 1656f4b..0a4a409 100644 --- a/src/main/java/com/att/research/mdbc/StateManager.java +++ b/src/main/java/com/att/research/mdbc/StateManager.java @@ -53,18 +53,22 @@ public class StateManager { private DatabasePartition ranges; public StateManager(String url, Properties info, DatabasePartition ranges, String sqlDatabase) throws MDBCServiceException { - this.sqlDatabase=sqlDatabase; - this.ranges=ranges; - this.url = url; - this.info = info; - this.transactionInfo = new TxCommitProgress(); - //\fixme this is not really used, delete! - String cassandraUrl = info.getProperty(Configuration.KEY_CASSANDRA_URL, Configuration.CASSANDRA_URL_DEFAULT); - String mixin = info.getProperty(Configuration.KEY_MUSIC_MIXIN_NAME, Configuration.MUSIC_MIXIN_DEFAULT); + this.sqlDatabase = sqlDatabase; + this.ranges = ranges; + this.url = url; + this.info = info; + this.transactionInfo = new TxCommitProgress(); + //\fixme this is not really used, delete! + String cassandraUrl = info.getProperty(Configuration.KEY_CASSANDRA_URL, Configuration.CASSANDRA_URL_DEFAULT); + String mixin = info.getProperty(Configuration.KEY_MUSIC_MIXIN_NAME, Configuration.MUSIC_MIXIN_DEFAULT); + init(mixin, cassandraUrl); + } + + protected void init(String mixin, String cassandraUrl) throws MDBCServiceException { this.musicManager = MixinFactory.createMusicInterface(mixin, cassandraUrl, info,ranges); this.musicManager.createKeyspace(); try { - this.musicManager.initializeMdbcDataStructures(); + this.musicManager.initializeMetricDataStructures(); } catch (MDBCServiceException e) { logger.error(EELFLoggerDelegate.errorLogger, e.getMessage(),AppMessages.UNKNOWNERROR, ErrorSeverity.CRITICAL, ErrorTypes.GENERALSERVICEERROR); throw(e); diff --git a/src/main/java/com/att/research/mdbc/configurations/NodeConfiguration.java b/src/main/java/com/att/research/mdbc/configurations/NodeConfiguration.java index 78850e3..d74dafb 100644 --- a/src/main/java/com/att/research/mdbc/configurations/NodeConfiguration.java +++ b/src/main/java/com/att/research/mdbc/configurations/NodeConfiguration.java @@ -22,8 +22,8 @@ public class NodeConfiguration { public DatabasePartition partition; public String nodeName; - public NodeConfiguration(String tables, String titIndex, String titTableName, String partitionId, String sqlDatabaseName, String node, String redoRecordsTable){ - partition = new DatabasePartition(toRanges(tables), titIndex, titTableName, partitionId, null, redoRecordsTable) ; + public NodeConfiguration(String tables, String mriIndex, String mriTableName, String partitionId, String sqlDatabaseName, String node, String redoRecordsTable){ + partition = new DatabasePartition(toRanges(tables), mriIndex, mriTableName, partitionId, null, redoRecordsTable) ; this.sqlDatabaseName = sqlDatabaseName; this.nodeName = node; } diff --git a/src/main/java/com/att/research/mdbc/configurations/TablesConfiguration.java b/src/main/java/com/att/research/mdbc/configurations/TablesConfiguration.java index 664520f..b86d058 100644 --- a/src/main/java/com/att/research/mdbc/configurations/TablesConfiguration.java +++ b/src/main/java/com/att/research/mdbc/configurations/TablesConfiguration.java @@ -19,7 +19,7 @@ import java.util.List; public class TablesConfiguration { private final String TIT_TABLE_NAME = "transactioninformation"; - private final String REDO_RECORDS_NAME = "redorecords"; + private final String MUSIC_TX_DIGEST_TABLE_NAME = "musictxdigest"; private transient static EELFLoggerDelegate logger = EELFLoggerDelegate.getLogger(TablesConfiguration.class); private List<PartitionInformation> partitions; @@ -54,14 +54,13 @@ public class TablesConfiguration { throw new MDBCServiceException("Partition was not correctly initialized"); } for(PartitionInformation partitionInfo : partitions){ - String titTableName = partitionInfo.titTableName; - titTableName = (titTableName==null || titTableName.isEmpty())?TIT_TABLE_NAME:titTableName; - //0) Create the corresponding TIT table - DatabaseOperations.CreateTransactionInformationTable(musicNamespace,titTableName); - String redoRecordsName = partitionInfo.rrtTableName; - redoRecordsName = (redoRecordsName==null || redoRecordsName.isEmpty())?REDO_RECORDS_NAME:redoRecordsName; - DatabaseOperations.CreateRedoRecordsTable(-1,musicNamespace,redoRecordsName); - //0) Create the corresponding TIT table + String mriTableName = partitionInfo.mriTableName; + mriTableName = (mriTableName==null || mriTableName.isEmpty())?TIT_TABLE_NAME:mriTableName; + //0) Create the corresponding Music Range Information table + DatabaseOperations.CreateMusicRangeInformationTable(musicNamespace,mriTableName); + String musicTxDigestTableName = partitionInfo.mtxdTableName; + musicTxDigestTableName = (musicTxDigestTableName==null || musicTxDigestTableName.isEmpty())? MUSIC_TX_DIGEST_TABLE_NAME :musicTxDigestTableName; + DatabaseOperations.CreateMusicTxDigest(-1,musicNamespace,musicTxDigestTableName); String partitionId; if(partitionInfo.partitionId==null || partitionInfo.partitionId.isEmpty()){ if(partitionInfo.replicationFactor==0){ @@ -76,9 +75,9 @@ public class TablesConfiguration { partitionId = partitionInfo.partitionId; } //2) Create a row in the transaction information table - String titIndex = DatabaseOperations.CreateEmptyTitRow(musicNamespace,titTableName,partitionId,null); + String mriTableIndex = DatabaseOperations.CreateEmptyTitRow(musicNamespace,mriTableName,partitionId,null); //3) Add owner and tit information to partition info table - RedoRow newRedoRow = new RedoRow(titTableName,titIndex); + RedoRow newRedoRow = new RedoRow(mriTableName,mriTableIndex); DatabaseOperations.updateRedoRow(musicNamespace,pitName,partitionId,newRedoRow,partitionInfo.owner,null); //4) Update ttp with the new partition for(String table: partitionInfo.tables) { @@ -87,7 +86,7 @@ public class TablesConfiguration { //5) Add it to the redo history table DatabaseOperations.createRedoHistoryBeginRow(musicNamespace,rhName,newRedoRow,partitionId,null); //6) Create config for this node - nodeConfigs.add(new NodeConfiguration(String.join(",",partitionInfo.tables),titIndex,titTableName,partitionId,sqlDatabaseName,partitionInfo.owner,redoRecordsName)); + nodeConfigs.add(new NodeConfiguration(String.join(",",partitionInfo.tables),mriTableIndex,mriTableName,partitionId,sqlDatabaseName,partitionInfo.owner,musicTxDigestTableName)); } return nodeConfigs; } @@ -124,8 +123,8 @@ public class TablesConfiguration { public class PartitionInformation{ private List<String> tables; private String owner; - private String titTableName; - private String rrtTableName; + private String mriTableName; + private String mtxdTableName; private String partitionId; private int replicationFactor; @@ -145,12 +144,12 @@ public class TablesConfiguration { this.owner = owner; } - public String getTitTableName() { - return titTableName; + public String getMriTableName() { + return mriTableName; } - public void setTitTableName(String titTableName) { - this.titTableName = titTableName; + public void setMriTableName(String mriTableName) { + this.mriTableName = mriTableName; } public String getPartitionId() { @@ -169,12 +168,12 @@ public class TablesConfiguration { this.replicationFactor = replicationFactor; } - public String getRrtTableName(){ - return rrtTableName; + public String getMtxdTableName(){ + return mtxdTableName; } - public void setRrtTableName(String rrtTableName) { - this.rrtTableName = rrtTableName; + public void setMtxdTableName(String mtxdTableName) { + this.mtxdTableName = mtxdTableName; } } } diff --git a/src/main/java/com/att/research/mdbc/configurations/ranges.json b/src/main/java/com/att/research/mdbc/configurations/ranges.json index afa343b..2a792e8 100644 --- a/src/main/java/com/att/research/mdbc/configurations/ranges.json +++ b/src/main/java/com/att/research/mdbc/configurations/ranges.json @@ -1,6 +1,6 @@ { - "transactionInformationTable": "transactioninformation", - "transactionInformationIndex": "d0e8ef2e-aeca-4261-8d9d-1679f560b85b", + "musicRangeInformationTable": "transactioninformation", + "musicRangeInformationIndex": "d0e8ef2e-aeca-4261-8d9d-1679f560b85b", "partitionId": "798110cf-9c61-4db2-9446-cb2dbab5a143", "lockId": "", "ranges": [ diff --git a/src/main/java/com/att/research/mdbc/configurations/tableConfiguration.json b/src/main/java/com/att/research/mdbc/configurations/tableConfiguration.json index b3c6224..e67dd0b 100644 --- a/src/main/java/com/att/research/mdbc/configurations/tableConfiguration.json +++ b/src/main/java/com/att/research/mdbc/configurations/tableConfiguration.json @@ -3,8 +3,8 @@ { "tables":["table11"], "owner":"", - "titTableName":"transactioninformation", - "rrtTableName":"redorecords", + "mriTableName":"musicrangeinformation", + "mtxdTableName":"musictxdigest", "partitionId":"", "replicationFactor":1 } diff --git a/src/main/java/com/att/research/mdbc/mixins/CassandraMixin.java b/src/main/java/com/att/research/mdbc/mixins/CassandraMixin.java index 033179a..5293682 100755 --- a/src/main/java/com/att/research/mdbc/mixins/CassandraMixin.java +++ b/src/main/java/com/att/research/mdbc/mixins/CassandraMixin.java @@ -27,7 +27,6 @@ import com.att.research.mdbc.tables.TxCommitProgress; import org.json.JSONObject; import org.onap.music.datastore.CassaLockStore; -import org.onap.music.datastore.CassaLockStore.LockObject; import org.onap.music.datastore.PreparedQueryObject; import org.onap.music.exceptions.MusicLockingException; import org.onap.music.exceptions.MusicQueryException; @@ -94,8 +93,8 @@ public class CassandraMixin implements MusicInterface { public static final String PARTITION_INFORMATION_TABLE_NAME = "partitioninfo"; public static final String REDO_HISTORY_TABLE_NAME= "redohistory"; //\TODO Add logic to change the names when required and create the tables when necessary - private String redoRecordTableName = "redorecords"; - private String transactionInformationTableName = "transactioninformation"; + private String musicTxDigestTableName = "musictxdigest"; + private String musicRangeInformationTableName = "musicrangeinformation"; private EELFLoggerDelegate logger = EELFLoggerDelegate.getLogger(CassandraMixin.class); @@ -162,7 +161,7 @@ public class CassandraMixin implements MusicInterface { this.music_ns = info.getProperty(KEY_MUSIC_NAMESPACE,DEFAULT_MUSIC_NAMESPACE); logger.info(EELFLoggerDelegate.applicationLogger,"MusicSqlManager: music_ns="+music_ns); - transactionInformationTableName = "transactioninformation"; + musicRangeInformationTableName = "musicrangeinformation"; createMusicKeyspace(); } @@ -221,10 +220,10 @@ public class CassandraMixin implements MusicInterface { } } @Override - public void initializeMdbcDataStructures() throws MDBCServiceException { + public void initializeMetricDataStructures() throws MDBCServiceException { try { - DatabaseOperations.CreateRedoRecordsTable(-1, music_ns, redoRecordTableName);//\TODO If we start partitioning the data base, we would need to use the redotable number - DatabaseOperations.CreateTransactionInformationTable(music_ns, transactionInformationTableName); + DatabaseOperations.CreateMusicTxDigest(-1, music_ns, musicTxDigestTableName);//\TODO If we start partitioning the data base, we would need to use the redotable number + DatabaseOperations.CreateMusicRangeInformationTable(music_ns, musicRangeInformationTableName); DatabaseOperations.CreateTableToPartitionTable(music_ns, TABLE_TO_PARTITION_TABLE_NAME); DatabaseOperations.CreatePartitionInfoTable(music_ns, PARTITION_INFORMATION_TABLE_NAME); DatabaseOperations.CreateRedoHistoryTable(music_ns, REDO_HISTORY_TABLE_NAME); @@ -1026,7 +1025,7 @@ public class CassandraMixin implements MusicInterface { } - private PreparedQueryObject createAppendRRTIndexToTitQuery(String titTable, String uuid, String table, String redoUuid){ + private PreparedQueryObject createAppendMtxdIndexToMriQuery(String titTable, String uuid, String table, String redoUuid){ PreparedQueryObject query = new PreparedQueryObject(); StringBuilder appendBuilder = new StringBuilder(); appendBuilder.append("UPDATE ") @@ -1065,7 +1064,7 @@ public class CassandraMixin implements MusicInterface { try { MusicCore.forciblyReleaseLock(fullyQualifiedKey,lockId); CassaLockStore lockingServiceHandle = MusicCore.getLockingServiceHandle(); - LockObject lockOwner = lockingServiceHandle.peekLockQueue(keyspace, table, key); + CassaLockStore.LockObject lockOwner = lockingServiceHandle.peekLockQueue(keyspace, table, key); while(lockOwner.lockRef != lockId) { MusicCore.forciblyReleaseLock(fullyQualifiedKey, lockOwner.lockRef); try { @@ -1096,12 +1095,12 @@ public class CassandraMixin implements MusicInterface { return lockId; } - protected void pushRowToRRT(String lockId, String commitId, HashMap<Range,StagingTable> transactionDigest) throws MDBCServiceException{ + protected void pushRowToMtxd(String lockId, String commitId, HashMap<Range,StagingTable> transactionDigest) throws MDBCServiceException{ PreparedQueryObject query = new PreparedQueryObject(); StringBuilder cqlQuery = new StringBuilder("INSERT INTO ") .append(music_ns) .append('.') - .append(redoRecordTableName) + .append(musicTxDigestTableName) .append(" (leaseid,leasecounter,transactiondigest) ") .append("VALUES ('") .append( lockId ).append("',") @@ -1123,15 +1122,15 @@ public class CassandraMixin implements MusicInterface { } } - protected void appendIndexToTit(String lockId, String commitId, String TITIndex) throws MDBCServiceException{ + protected void appendIndexToMri(String lockId, String commitId, String MriIndex) throws MDBCServiceException{ StringBuilder redoUuidBuilder = new StringBuilder(); redoUuidBuilder.append("('") .append(lockId) .append("',") .append(commitId) .append(")"); - PreparedQueryObject appendQuery = createAppendRRTIndexToTitQuery(transactionInformationTableName, TITIndex, redoRecordTableName, redoUuidBuilder.toString()); - ReturnType returnType = MusicCore.criticalPut(music_ns, transactionInformationTableName, TITIndex, appendQuery, lockId, null); + PreparedQueryObject appendQuery = createAppendMtxdIndexToMriQuery(musicRangeInformationTableName, MriIndex, musicTxDigestTableName, redoUuidBuilder.toString()); + ReturnType returnType = MusicCore.criticalPut(music_ns, musicRangeInformationTableName, MriIndex, appendQuery, lockId, null); if(returnType.getResult().compareTo(ResultType.SUCCESS) != 0 ){ logger.error(EELFLoggerDelegate.errorLogger, "Error when executing append operation with return type: "+returnType.getMessage()); throw new MDBCServiceException("Error when executing append operation with return type: "+returnType.getMessage()); @@ -1140,16 +1139,16 @@ public class CassandraMixin implements MusicInterface { @Override public void commitLog(DBInterface dbi, DatabasePartition partition, HashMap<Range,StagingTable> transactionDigest, String txId ,TxCommitProgress progressKeeper) throws MDBCServiceException{ - String TITIndex = partition.getTransactionInformationIndex(); - if(TITIndex.isEmpty()) { - //\TODO Fetch TITIndex from the Range Information Table + String MriIndex = partition.getMusicRangeInformationIndex(); + if(MriIndex.isEmpty()) { + //\TODO Fetch MriIndex from the Range Information Table throw new MDBCServiceException("TIT Index retrieval not yet implemented"); } - String fullyQualifiedTitKey = music_ns+"."+ transactionInformationTableName +"."+TITIndex; + String fullyQualifiedTitKey = music_ns+"."+ musicRangeInformationTableName +"."+MriIndex; //0. See if reference to lock was already created String lockId = partition.getLockId(); if(lockId == null || lockId.isEmpty()) { - lockId = createAndAssignLock(fullyQualifiedTitKey,partition,music_ns,transactionInformationTableName,TITIndex); + lockId = createAndAssignLock(fullyQualifiedTitKey,partition,music_ns, musicRangeInformationTableName,MriIndex); } String commitId; @@ -1164,14 +1163,14 @@ public class CassandraMixin implements MusicInterface { //Add creation type of transaction digest //1. Push new row to RRT and obtain its index - pushRowToRRT(lockId, commitId, transactionDigest); + pushRowToMtxd(lockId, commitId, transactionDigest); //2. Save RRT index to RQ if(progressKeeper!= null) { progressKeeper.setRecordId(txId,new RedoRecordId(lockId, commitId)); } //3. Append RRT index into the corresponding TIT row array - appendIndexToTit(lockId,commitId,TITIndex); + appendIndexToMri(lockId,commitId,MriIndex); } /** diff --git a/src/main/java/com/att/research/mdbc/mixins/MusicInterface.java b/src/main/java/com/att/research/mdbc/mixins/MusicInterface.java index 9051ab6..6e2e0ca 100755 --- a/src/main/java/com/att/research/mdbc/mixins/MusicInterface.java +++ b/src/main/java/com/att/research/mdbc/mixins/MusicInterface.java @@ -31,7 +31,7 @@ public interface MusicInterface { * This function is used to created all the required data structures, both local * \TODO Check if this function is required in the MUSIC interface or could be just created on the constructor */ - void initializeMdbcDataStructures() throws MDBCServiceException; + void initializeMetricDataStructures() throws MDBCServiceException; /** * Get the name of this MusicInterface mixin object. * @return the name diff --git a/src/main/java/com/att/research/mdbc/mixins/MusicMixin.java b/src/main/java/com/att/research/mdbc/mixins/MusicMixin.java index bf865b2..43f36e3 100644 --- a/src/main/java/com/att/research/mdbc/mixins/MusicMixin.java +++ b/src/main/java/com/att/research/mdbc/mixins/MusicMixin.java @@ -181,7 +181,7 @@ public class MusicMixin implements MusicInterface { } @Override - public void initializeMdbcDataStructures() { + public void initializeMetricDataStructures() { // } diff --git a/src/main/java/com/att/research/mdbc/tools/CreatePartition.java b/src/main/java/com/att/research/mdbc/tools/CreatePartition.java index 09524cb..a38274b 100644 --- a/src/main/java/com/att/research/mdbc/tools/CreatePartition.java +++ b/src/main/java/com/att/research/mdbc/tools/CreatePartition.java @@ -1,17 +1,10 @@ package com.att.research.mdbc.tools; import com.att.research.logging.EELFLoggerDelegate; -import com.att.research.mdbc.DatabasePartition; -import com.att.research.mdbc.MDBCUtils; -import com.att.research.mdbc.Range; import com.att.research.mdbc.configurations.NodeConfiguration; import com.beust.jcommander.JCommander; import com.beust.jcommander.Parameter; -import java.io.IOException; -import java.util.HashSet; -import java.util.Set; - public class CreatePartition { public static final EELFLoggerDelegate LOG = EELFLoggerDelegate.getLogger(CreatePartition.class); @@ -21,15 +14,15 @@ public class CreatePartition { @Parameter(names = { "-f", "--file" }, required = true, description = "This is the output file that is going to have the configuration for the ranges") private String file; - @Parameter(names = { "-i", "--tit-index" }, required = true, - description = "Index in the TiT Table") - private String titIndex; - @Parameter(names = { "-n", "--tit-table-name" }, required = true, - description = "Tit Table name") - private String titTable; - @Parameter(names = { "-r", "--redorecords-table-name" }, required = true, - description = "Redo Records Table name") - private String rrTable; + @Parameter(names = { "-i", "--mri-index" }, required = true, + description = "Index in the Mri Table") + private String mriIndex; + @Parameter(names = { "-m", "--mri-table-name" }, required = true, + description = "Mri Table name") + private String mriTable; + @Parameter(names = { "-r", "--music-tx-digest-table-name" }, required = true, + description = "Music Transaction Digest Table name") + private String mtxdTable; @Parameter(names = { "-p", "--partition-id" }, required = true, description = "Partition Id") private String partitionId; @@ -43,7 +36,7 @@ public class CreatePartition { } public void convert(){ - config = new NodeConfiguration(tables,titIndex,titTable,partitionId,"test","",rrTable); + config = new NodeConfiguration(tables, mriIndex,mriTable,partitionId,"test","", mtxdTable); } public void saveToFile(){ |