Apache Jena: Administrative users can create files outside the server directory space via the admin UI
Description
Users with administrator access can create databases files outside the files area of the Fuseki server.
This issue affects Apache Jena version up to 5.4.0.
Users are recommended to upgrade to version 5.5.0, which fixes the issue.
AI Insight
LLM-synthesized narrative grounded in this CVE's description and references.
Administrator users in Apache Jena Fuseki could create database files outside the intended files area, leading to potential unauthorized file system access.
Vulnerability
Overview
CVE-2025-49656 describes a path traversal vulnerability in Apache Jena Fuseki, a SPARQL server. The root cause is insufficient validation of request parameters when administrator users create new databases. This allows an admin to specify a file path outside the intended Fuseki files area, leading to arbitrary file creation on the server's file system [1][2].
Exploitation
Exploitation requires administrator-level access to the Fuseki server. An attacker with such privileges can craft API requests to create a database with a path that escapes the designated base directory. The vulnerability exists in versions up to and including 5.4.0 [1].
Impact
A successful attack enables the attacker to write database files to arbitrary locations on the server. This could lead to overwriting critical system files, planting malicious configuration files, or gaining further unauthorized access to the host system [1].
Mitigation
The issue is fixed in Apache Jena version 5.5.0. The fix includes validation of request parameters in the ActionDatasets class and introduces a system property fuseki:allowAddByConfigFile to control config file uploads [2][3]. Users are strongly recommended to upgrade to the latest version [1].
AI Insight generated on May 19, 2026. Synthesized from this CVE's description and the cited reference URLs; citations are validated against the source bundle.
Affected packages
Versions sourced from the GitHub Security Advisory.
| Package | Affected versions | Patched versions |
|---|---|---|
org.apache.jena:jena-fusekiMaven | < 5.5.0 | 5.5.0 |
Affected products
2- Apache Software Foundation/Apache Jenav5Range: 0
Patches
203c5265910aaGH-3288: Update jena-fuseki-webapp
7 files changed · +1356 −553
jena-fuseki2/jena-fuseki-webapp/src/main/java/org/apache/jena/fuseki/mgt/ActionDatasets.java+346 −164 modified@@ -19,17 +19,11 @@ package org.apache.jena.fuseki.mgt; import static java.lang.String.format; -import static org.apache.jena.atlas.lib.Lib.lowercase; -import java.io.IOException; -import java.io.OutputStream; import java.io.StringReader; import java.nio.file.Files; import java.nio.file.Path; -import java.util.HashMap; -import java.util.List; -import java.util.Map; -import java.util.UUID; +import java.util.*; import jakarta.servlet.http.HttpServletRequest; import org.apache.commons.lang3.StringUtils; @@ -39,51 +33,58 @@ import org.apache.jena.atlas.json.JsonValue; import org.apache.jena.atlas.lib.FileOps; import org.apache.jena.atlas.lib.InternalErrorException; +import org.apache.jena.atlas.lib.NotImplemented; import org.apache.jena.atlas.logging.FmtLog; import org.apache.jena.atlas.web.ContentType; import org.apache.jena.datatypes.xsd.XSDDatatype; +import org.apache.jena.dboe.base.file.Location; +import org.apache.jena.fuseki.FusekiConfigException; import org.apache.jena.fuseki.build.DatasetDescriptionMap; import org.apache.jena.fuseki.build.FusekiConfig; import org.apache.jena.fuseki.ctl.ActionContainerItem; import org.apache.jena.fuseki.ctl.JsonDescription; -import org.apache.jena.fuseki.server.DataAccessPoint; -import org.apache.jena.fuseki.server.DataService; -import org.apache.jena.fuseki.server.FusekiVocab; -import org.apache.jena.fuseki.server.ServerConst; +import org.apache.jena.fuseki.metrics.MetricsProvider; +import org.apache.jena.fuseki.server.*; import org.apache.jena.fuseki.servlets.ActionLib; import org.apache.jena.fuseki.servlets.HttpAction; import org.apache.jena.fuseki.servlets.ServletOps; -import org.apache.jena.fuseki.system.DataUploader; import org.apache.jena.fuseki.system.FusekiNetLib; import org.apache.jena.fuseki.webapp.FusekiWebapp; +import org.apache.jena.graph.Graph; +import org.apache.jena.graph.Node; +import org.apache.jena.query.Query; +import org.apache.jena.query.QueryFactory; import org.apache.jena.rdf.model.*; +import org.apache.jena.rdf.model.impl.Util; import org.apache.jena.riot.*; import org.apache.jena.riot.system.StreamRDF; import org.apache.jena.riot.system.StreamRDFLib; +import org.apache.jena.shared.JenaException; +import org.apache.jena.sparql.core.DatasetGraph; +import org.apache.jena.sparql.core.Quad; import org.apache.jena.sparql.core.assembler.AssemblerUtils; +import org.apache.jena.sparql.exec.QueryExec; +import org.apache.jena.sparql.exec.RowSet; import org.apache.jena.sparql.util.FmtUtils; +import org.apache.jena.system.G; +import org.apache.jena.tdb1.TDB1; +import org.apache.jena.tdb2.TDB2; import org.apache.jena.vocabulary.RDF; import org.apache.jena.web.HttpSC; public class ActionDatasets extends ActionContainerItem { - static private Property pServiceName = FusekiVocab.pServiceName; - //static private Property pStatus = FusekiVocab.pStatus; private static final String paramDatasetName = "dbName"; private static final String paramDatasetType = "dbType"; - private static final String tDatabaseTDB = "tdb"; - private static final String tDatabaseTDB1 = "tdb1"; + private static final String tDatabaseTDB1 = "tdb"; private static final String tDatabaseTDB2 = "tdb2"; private static final String tDatabaseMem = "mem"; - // Sync lock. - private static final Object lock = new Object(); - public ActionDatasets() { super(); } @Override - public void validate(HttpAction action) {} + public void validate(HttpAction action) { } // ---- GET : return details of dataset or datasets. @Override @@ -111,10 +112,10 @@ protected JsonValue execGetItem(HttpAction action) { // ---- POST + /** Create dataset */ @Override protected JsonValue execPostContainer(HttpAction action) { UUID uuid = UUID.randomUUID(); - DatasetDescriptionMap registry = new DatasetDescriptionMap(); ContentType ct = ActionLib.getContentType(action); @@ -123,118 +124,223 @@ protected JsonValue execPostContainer(HttpAction action) { if ( ct == null && ! hasParams ) ServletOps.errorBadRequest("Bad request - Content-Type or both parameters dbName and dbType required"); - boolean committed = false; - // Also acts as a concurrency lock - synchronized(lock) { - String systemFileCopy = null; - String configFile = null; + boolean succeeded = false; + // Used in clear-up. + String configFile = null; + String systemFileCopy = null; - try { - // Where to build the templated service/database. - Model modelData = ModelFactory.createDefaultModel(); - StreamRDF dest = StreamRDFLib.graph(modelData.getGraph()); - - if ( hasParams || WebContent.isHtmlForm(ct) ) - assemblerFromForm(action, dest); - else if ( WebContent.isMultiPartForm(ct) ) - assemblerFromUpload(action, dest); - else - assemblerFromBody(action, dest); + DatasetDescriptionMap registry = new DatasetDescriptionMap(); - // ---- - // Keep a persistent copy immediately. This is not used for - // anything other than being "for the record". - systemFileCopy = FusekiWebapp.dirSystemFileArea.resolve(uuid.toString()).toString(); - try ( OutputStream outCopy = IO.openOutputFile(systemFileCopy) ) { - RDFDataMgr.write(outCopy, modelData, Lang.TURTLE); + synchronized (FusekiWebapp.systemLock) { + try { + // Get the request input. + Model modelFromRequest = ModelFactory.createDefaultModel(); + StreamRDF dest = StreamRDFLib.graph(modelFromRequest.getGraph()); + + boolean templatedRequest = false; + + try { + if ( hasParams || WebContent.isHtmlForm(ct) ) { + assemblerFromForm(action, dest); + templatedRequest = true; + // dbName, dbType + } else if ( WebContent.isMultiPartForm(ct) ) { + // Cannot be enabled. + ServletOps.errorBadRequest("Service configuration from a multipart upload not supported"); + //assemblerFromUpload(action, dest); + } else { + if ( ! FusekiWebapp.allowConfigFiles() ) + ServletOps.errorBadRequest("Service configuration from an upload file not supported"); + assemblerFromBody(action, dest); + } + } catch (RiotException ex) { + ActionLib.consumeBody(action); + action.log.warn(format("[%d] Failed to read configuration: %s", action.id, ex.getMessage())); + ServletOps.errorBadRequest("Failed to read configuration"); } + // ---- + // Add the dataset and graph wiring for assemblers Model model = ModelFactory.createDefaultModel(); - model.add(modelData); - // Add dataset and model declarations. + model.add(modelFromRequest); model = AssemblerUtils.prepareForAssembler(model); // ---- // Process configuration. - // Returns the "service fu:name NAME" statement Statement stmt = findService(model); + if ( stmt == null ) { + action.log.warn(format("[%d] No service name", action.id)); + ServletOps.errorBadRequest(format("No service name")); + } Resource subject = stmt.getSubject(); Literal object = stmt.getObject().asLiteral(); if ( object.getDatatype() != null && ! object.getDatatype().equals(XSDDatatype.XSDstring) ) action.log.warn(format("[%d] Service name '%s' is not a string", action.id, FmtUtils.stringForRDFNode(object))); - String datasetPath; - { // Check the name provided. + final String datasetPath; + { String datasetName = object.getLexicalForm(); // This duplicates the code FusekiBuilder.buildDataAccessPoint to give better error messages and HTTP status code." // ---- Check and canonicalize name. - if ( datasetName.isEmpty() ) - ServletOps.error(HttpSC.BAD_REQUEST_400, "Empty dataset name"); - if ( StringUtils.isBlank(datasetName) ) - ServletOps.error(HttpSC.BAD_REQUEST_400, format("Whitespace dataset name: '%s'", datasetName)); - if ( datasetName.contains(" ") ) - ServletOps.error(HttpSC.BAD_REQUEST_400, format("Bad dataset name (contains spaces) '%s'",datasetName)); - if ( datasetName.equals("/") ) - ServletOps.error(HttpSC.BAD_REQUEST_400, format("Bad dataset name '%s'",datasetName)); + // Various explicit check for better error messages. + + if ( datasetName.isEmpty() ) { + action.log.warn(format("[%d] Empty dataset name", action.id)); + ServletOps.errorBadRequest("Empty dataset name"); + } + if ( StringUtils.isBlank(datasetName) ) { + action.log.warn(format("[%d] Whitespace dataset name: '%s'", action.id, datasetName)); + ServletOps.errorBadRequest(format("Whitespace dataset name: '%s'", datasetName)); + } + if ( datasetName.contains(" ") ) { + action.log.warn(format("[%d] Bad dataset name (contains spaces) '%s'", action.id, datasetName)); + ServletOps.errorBadRequest(format("Bad dataset name (contains spaces) '%s'", datasetName)); + } + if ( datasetName.equals("/") ) { + action.log.warn(format("[%d] Bad dataset name '%s'", action.id, datasetName)); + ServletOps.errorBadRequest(format("Bad dataset name '%s'", datasetName)); + } + + // The service names must be a valid URI path + try { + ValidString validServiceName = Validators.serviceName(datasetName); + } catch (FusekiConfigException ex) { + action.log.warn(format("[%d] Invalid service name: '%s'", action.id, datasetName)); + ServletOps.error(HttpSC.BAD_REQUEST_400, format("Invalid service name: '%s'", datasetName)); + } + + // Canonical - starts with "/",does not end in "/" datasetPath = DataAccessPoint.canonical(datasetName); - // ---- Check whether it already exists - if ( action.getDataAccessPointRegistry().isRegistered(datasetPath) ) - // And abort. - ServletOps.error(HttpSC.CONFLICT_409, "Name already registered "+datasetPath); + + // For this operation, check additionally that the path does not go outside the expected file area. + // This imposes the path component-only rule and does not allow ".." + if ( ! isValidServiceName(datasetPath) ) { + action.log.warn(format("[%d] Database service name not acceptable: '%s'", action.id, datasetName)); + ServletOps.error(HttpSC.BAD_REQUEST_400, format("Database service name not acceptable: '%s'", datasetName)); + } } + // ---- Check whether it already exists + if ( action.getDataAccessPointRegistry().isRegistered(datasetPath) ) { + action.log.warn(format("[%d] Name already registered '%s'", action.id, datasetPath)); + ServletOps.error(HttpSC.CONFLICT_409, format("Name already registered '%s'", datasetPath)); + } + + // -- Validate any TDB locations. + // If this is a templated request, there is no need to do this + // because the location is "datasetPath" which has been checked. + if ( ! templatedRequest ) { + // -- Validate any TDB locations. + // If this is a templated request, there is no need to do this because the location is "datasetPath" + List<String> tdbLocations = tdbLocations(action, model.getGraph()); + for(String tdbLocation : tdbLocations ) { + if ( ! isValidTDBLocation(tdbLocation) ) { + action.log.warn(format("[%d] TDB database location not acceptable: '%s'", action.id, tdbLocation)); + ServletOps.error(HttpSC.BAD_REQUEST_400, format("TDB database location not acceptable: '%s'", tdbLocation)); + } + } + } + + // ---- + // Keep a persistent copy with a globally unique name. + // This is not used for anything other than being "for the record". + systemFileCopy = FusekiWebapp.dirSystemFileArea.resolve(uuid.toString()).toString(); + RDFWriter.source(model).lang(Lang.TURTLE).output(systemFileCopy); + + // ---- action.log.info(format("[%d] Create database : name = %s", action.id, datasetPath)); - configFile = FusekiWebapp.generateConfigurationFilename(datasetPath); List<String> existing = FusekiWebapp.existingConfigurationFile(datasetPath); if ( ! existing.isEmpty() ) ServletOps.error(HttpSC.CONFLICT_409, "Configuration file for '"+datasetPath+"' already exists"); - // Write to configuration directory. - try ( OutputStream outCopy = IO.openOutputFile(configFile) ) { - RDFDataMgr.write(outCopy, modelData, Lang.TURTLE); - } + configFile = FusekiWebapp.generateConfigurationFilename(datasetPath); + // ---- Build the service DataAccessPoint dataAccessPoint = FusekiConfig.buildDataAccessPoint(subject.getModel().getGraph(), subject.asNode(), registry); if ( dataAccessPoint == null ) { FmtLog.error(action.log, "Failed to build DataAccessPoint: datasetPath = %s; DataAccessPoint name = %s", datasetPath, dataAccessPoint); ServletOps.errorBadRequest("Failed to build DataAccessPoint"); return null; } dataAccessPoint.getDataService().setEndpointProcessors(action.getOperationRegistry()); - dataAccessPoint.getDataService().goActive(); + + // Write to configuration directory. + RDFWriter.source(model).lang(Lang.TURTLE).output(configFile); + if ( ! datasetPath.equals(dataAccessPoint.getName()) ) FmtLog.warn(action.log, "Inconsistent names: datasetPath = %s; DataAccessPoint name = %s", datasetPath, dataAccessPoint); + dataAccessPoint.getDataService().goActive(); + succeeded = true; + + // At this point, a server restarting will find the new service. + // This next line makes it dispatchable in this running server. action.getDataAccessPointRegistry().register(dataAccessPoint); - action.setResponseContentType(WebContent.contentTypeTextPlain); - ServletOps.success(action); - committed = true; + // Add to metrics + MetricsProvider metricProvider = action.getMetricsProvider(); + if ( metricProvider != null ) + action.getMetricsProvider().addDataAccessPointMetrics(dataAccessPoint); - } catch (IOException ex) { IO.exception(ex); } - finally { - if ( ! committed ) { + action.setResponseContentType(WebContent.contentTypeTextPlain); + ServletOps.success(action); + } finally { + // Clear-up on failure. + if ( ! succeeded ) { if ( systemFileCopy != null ) FileOps.deleteSilent(systemFileCopy); if ( configFile != null ) FileOps.deleteSilent(configFile); - } - } + return null; } - return null; + } + + /** + * Check whether a service name is acceptable. + * A service name is used as a filesystem path component, + * except it may have a leading "/"., to store the database and the configuration. + * <p> + * The canonical name for a service (see {@link DataAccessPoint#canonical}) + * starts with a "/" and this will be added if necessary. + */ + private boolean isValidServiceName(String datasetPath) { + // Leading "/" is OK , nowhere else is. + int idx = datasetPath.indexOf('/', 1); + if ( idx > 0 ) + return false; + // No slash, except maybe at the start so a meaningful use of .. can only be at the start. + if ( datasetPath.startsWith("/..")) + return false; + // Character restrictions done by Validators.serviceName + return true; + } + + // This works for TDB1 as well. + private boolean isValidTDBLocation(String tdbLocation) { + Location location = Location.create(tdbLocation); + if ( location.isMem() ) + return true; + String locationString = location.getDirectoryPath(); + // No ".." + if (locationString.startsWith("..") || locationString.contains("/..") ) { + // That test was too strict. + List<String> components = FileOps.pathComponents(locationString); + if ( components.contains("..") ) + return false; + } + return true; } /** Find the service resource. There must be only one in the configuration. */ private Statement findService(Model model) { // Try to find by unique pServiceName (max backwards compatibility) // then try to find by rdf:type fuseki:Service. - // JENA-1794 Statement stmt = getOne(model, null, pServiceName, null); // null means 0 or many, not one. @@ -262,16 +368,17 @@ private Statement findService(Model model) { stmt = stmt3; } + if ( stmt == null ) + return null; + if ( ! stmt.getObject().isLiteral() ) - ServletOps.errorBadRequest("Found "+FmtUtils.stringForRDFNode(stmt.getObject())+" : Service names are strings, then used to build the external URI"); + ServletOps.errorBadRequest("Found "+FmtUtils.stringForRDFNode(stmt.getObject())+" : Service names are strings, which are then used to build the external URI"); return stmt; } @Override protected JsonValue execPostItem(HttpAction action) { - // This used to be the state change function -- active/inactive. - // Leave the core of the operation - tests used it to ping for a database. String name = getItemDatasetName(action); if ( name == null ) name = "''"; @@ -282,6 +389,15 @@ protected JsonValue execPostItem(HttpAction action) { if ( dap == null ) ServletOps.errorNotFound("Not found: dataset "+name); + + DataService dSrv = dap.getDataService(); + if ( dSrv == null ) + // If not set explicitly, take from DataAccessPoint + dSrv = action.getDataAccessPoint().getDataService(); + + String s = action.getRequestParameter("state"); + if ( s == null || s.isEmpty() ) + ServletOps.errorBadRequest("No state change given"); return null; } @@ -298,124 +414,147 @@ protected void execDeleteItem(HttpAction action) { if ( ! action.getDataAccessPointRegistry().isRegistered(name) ) ServletOps.errorNotFound("No such dataset registered: "+name); - boolean committed = false; - synchronized(lock) { - // Redo check inside transaction. - DataAccessPoint ref = action.getDataAccessPointRegistry().get(name); - if ( ref == null ) - ServletOps.errorNotFound("No such dataset registered: "+name); - - // Get a reference before removing. - DataService dataService = ref.getDataService(); - // ---- Make it invisible in this running server. - action.getDataAccessPointRegistry().remove(name); - - // Find the configuration. - String filename = name.startsWith("/") ? name.substring(1) : name; - List<String> configurationFiles = FusekiWebapp.existingConfigurationFile(filename); - - if ( configurationFiles.isEmpty() ) { - // ---- Unmanaged - action.log.warn(format("[%d] Can't delete database configuration - not a managed database; dataset=%s", action.id, name)); + boolean succeeded = false; + + synchronized(FusekiWebapp.systemLock) { + try { + // Redo check inside transaction. + DataAccessPoint ref = action.getDataAccessPointRegistry().get(name); + if ( ref == null ) + ServletOps.errorNotFound("No such dataset registered: "+name); + + // Get a reference before removing. + DataService dataService = ref.getDataService(); + + // Remove from the registry - operation dispatch will not find it any more. + action.getDataAccessPointRegistry().remove(name); + + // Find the configuration. + List<String> configurationFiles = FusekiWebapp.existingConfigurationFile(name); + + if ( configurationFiles.isEmpty() ) { + // -- Unmanaged + action.log.warn(format("[%d] Can't delete database configuration - not a managed database", action.id, name)); // ServletOps.errorOccurred(format("Can't delete database - not a managed configuration", name)); - committed = true; - ServletOps.success(action); - return; - } + succeeded = true; + ServletOps.success(action); + return; + } - if ( configurationFiles.size() > 1 ) { - // -- This should not happen. - action.log.warn(format("[%d] There are %d configuration files, not one.", action.id, configurationFiles.size())); - ServletOps.errorOccurred(format("There are %d configuration files, not one. Delete not performed; manual clean up of the filesystem needed.", - configurationFiles.size())); - return; - } + if ( configurationFiles.size() > 1 ) { + // -- This should not happen. + action.log.warn(format("[%d] There are %d configuration files, not one.", action.id, configurationFiles.size())); + ServletOps.errorOccurred(format("There are %d configuration files, not one. Delete not performed; manual clean up of the filesystem needed.", + configurationFiles.size())); + return; + } - // ---- Remove managed database. - String cfgPathname = configurationFiles.get(0); + // -- Remove managed database. + String cfgPathname = configurationFiles.get(0); - // Delete configuration file. - // Once deleted, server restart will not have the database. - FileOps.deleteSilent(cfgPathname); + // Delete configuration file. + // Once deleted, server restart will not have the database. + FileOps.deleteSilent(cfgPathname); - // Delete the database for real only when it is in the server "run/databases" - // area. Don't delete databases that reside elsewhere. We do delete the - // configuration file, so the databases will not be associated with the server - // anymore. + // Delete the database for real only if it is in the server + // "run/databases" area. Don't delete databases that reside + // elsewhere. We have already deleted the configuration file, so the + // databases will not be associated with the server anymore. - @SuppressWarnings("removal") - boolean isTDB1 = org.apache.jena.tdb1.sys.TDBInternal.isTDB1(dataService.getDataset()); - boolean isTDB2 = org.apache.jena.tdb2.sys.TDBInternal.isTDB2(dataService.getDataset()); + @SuppressWarnings("removal") + boolean isTDB1 = org.apache.jena.tdb1.sys.TDBInternal.isTDB1(dataService.getDataset()); + boolean isTDB2 = org.apache.jena.tdb2.sys.TDBInternal.isTDB2(dataService.getDataset()); - // This occasionally fails in tests due to outstanding transactions. - // Unclear what's holding the transaction (maybe another test clearing up slowly). - try { - dataService.shutdown(); - } catch (/*DBOE*/ Exception ex) { } - // JENA-1481: Really delete files. - if ( ( isTDB1 || isTDB2 ) ) { - // Delete databases created by the UI, or the admin operation, which are - // in predictable, unshared location on disk. - // There may not be any database files, the in-memory case. - Path pDatabase = FusekiWebapp.dirDatabases.resolve(filename); - if ( Files.exists(pDatabase)) { - try { - if ( Files.isSymbolicLink(pDatabase)) { - action.log.info(format("[%d] Database is a symbolic link, not removing files %s", action.id, pDatabase)); - } else { - IO.deleteAll(pDatabase); - action.log.info(format("[%d] Deleted database files %s", action.id, pDatabase)); + try { + dataService.shutdown(); + } catch (JenaException ex) { + return; + } + // JENA-1481: Really delete files. + if ( ( isTDB1 || isTDB2 ) ) { + // Delete databases created by the UI, or the admin operation, which are + // in predictable, unshared locations on disk. + // There may not be any database files, the in-memory case. + // (TDB supports an in-memory mode.) + String filename = name.startsWith("/") ? name.substring(1) : name; + Path pDatabase = FusekiWebapp.dirDatabases.resolve(filename); + if ( Files.exists(pDatabase)) { + try { + if ( Files.isSymbolicLink(pDatabase)) { + action.log.info(format("[%d] Database is a symbolic link, not removing files", action.id, pDatabase)); + } else { + IO.deleteAll(pDatabase); + action.log.info(format("[%d] Deleted database files %s", action.id, pDatabase)); + } + } catch (RuntimeIOException ex) { + action.log.error(format("[%d] Error while deleting database files %s: %s", action.id, pDatabase, ex.getMessage()), ex); + // But we have managed to remove it from the running server, and removed its configuration, so declare victory. } - } catch (RuntimeIOException ex) { - action.log.error(format("[%d] Error while deleting database files %s: %s", action.id, pDatabase, ex.getMessage()), ex); - // But we have managed to remove it from the running server, and removed its configuration, so declare victory. } } + + succeeded = true; + ServletOps.success(action); + } finally { + // No clearup needed } - ServletOps.success(action); } } private static void assemblerFromBody(HttpAction action, StreamRDF dest) { bodyAsGraph(action, dest); } - private static Map<String, String> dbTypeToTemplate = Map.of( - // Default TDB - tDatabaseTDB, Template.templateTDB2_FN, - // Specific TDB - tDatabaseTDB1, Template.templateTDB1_FN, - tDatabaseTDB2, Template.templateTDB2_FN, - // Transactional, in-memory - tDatabaseMem, Template.templateTIM_MemFN); + private static Map<String, String> dbTypeToTemplate = new HashMap<>(); + static { + dbTypeToTemplate.put(tDatabaseTDB1, Template.templateTDB1_FN); + dbTypeToTemplate.put(tDatabaseTDB2, Template.templateTDB2_FN); + dbTypeToTemplate.put(tDatabaseMem, Template.templateTIM_MemFN); + } private static void assemblerFromForm(HttpAction action, StreamRDF dest) { - String x = action.getRequestQueryString(); String dbType = action.getRequestParameter(paramDatasetType); String dbName = action.getRequestParameter(paramDatasetName); - if ( StringUtils.isBlank(dbType) || StringUtils.isBlank(dbName) ) - ServletOps.errorBadRequest("Received HTML form. Both parameters 'dbName' and 'dbType' required"); + // Test for null, empty or only whitespace. + if ( StringUtils.isBlank(dbType) || StringUtils.isBlank(dbName) ) { + action.log.warn(format("[%d] Both parameters 'dbName' and 'dbType' required and not be blank", action.id)); + ServletOps.errorBadRequest("Received HTML form. Both parameters 'dbName' and 'dbType' required"); + } Map<String, String> params = new HashMap<>(); - if ( dbName.startsWith("/") ) params.put(Template.NAME, dbName.substring(1)); else params.put(Template.NAME, dbName); + params.put(Template.NAME, dbName); FusekiWebapp.addGlobals(params); - //action.log.info(format("[%d] Create database : name = %s, type = %s", action.id, dbName, dbType )); - - String template = dbTypeToTemplate.get(lowercase(dbType)); - if ( template == null ) - ServletOps.errorBadRequest(format("dbType can be only '%s' ('%s','%s') or '%s'", tDatabaseTDB, tDatabaseTDB1, tDatabaseTDB2, tDatabaseMem)); + String template = dbTypeToTemplate.get(dbType.toLowerCase(Locale.ROOT)); + if ( template == null ) { + List<String> keys = new ArrayList<>(dbTypeToTemplate.keySet()); + Collections.sort(keys); + ServletOps.errorBadRequest(format("dbType can be only one of %s", keys)); + } - String syntax = TemplateFunctions.templateFile(template, params, Lang.TTL); - RDFParser.create().source(new StringReader(syntax)).base("http://base/").lang(Lang.TTL).parse(dest); + String instance = TemplateFunctions.templateFile(template, params, Lang.TTL); + RDFParser.create().source(new StringReader(instance)).base("http://base/").lang(Lang.TTL).parse(dest); } private static void assemblerFromUpload(HttpAction action, StreamRDF dest) { - DataUploader.incomingData(action, dest); + throw new NotImplemented(); + //DataUploader.incomingData(action, dest); + } + + // ---- Auxiliary functions + + private static Quad getOne(DatasetGraph dsg, Node g, Node s, Node p, Node o) { + Iterator<Quad> iter = dsg.findNG(g, s, p, o); + if ( ! iter.hasNext() ) + return null; + Quad q = iter.next(); + if ( iter.hasNext() ) + return null; + return q; } private static Statement getOne(Model m, Resource s, Property p, RDFNode o) { @@ -438,6 +577,49 @@ private static void bodyAsGraph(HttpAction action, StreamRDF dest) { return; } dest.prefix("root", base+"#"); - ActionLib.parseOrError(action, dest, lang, base); + ActionLib.parse(action, dest, lang, base); + } + + // ---- POST + + private static final String NL = "\n"; + + @SuppressWarnings("removal") + private static final String queryStringLocations = + "PREFIX tdb1: <"+TDB1.namespace+">"+NL+ + "PREFIX tdb2: <"+TDB2.namespace+">"+NL+ + """ + SELECT * { + ?x ( tdb2:location | tdb1:location) ?location + } + """ ; + + private static final Query queryLocations = QueryFactory.create(queryStringLocations); + + private static List<String> tdbLocations(HttpAction action, Graph configGraph) { + try ( QueryExec exec = QueryExec.graph(configGraph).query(queryLocations).build() ) { + RowSet results = exec.select(); + List<String> locations = new ArrayList<>(); + results.forEach(b->{ + Node loc = b.get("location"); + String location; + if ( loc.isURI() ) + location = loc.getURI(); + else if ( Util.isSimpleString(loc) ) + location = G.asString(loc); + else { + //action.log.warn(format("[%d] Database location is not a string nor a URI", action.id)); + // No return + ServletOps.errorBadRequest("TDB database location is not a string"); + location = null; + } + locations.add(location); + }); + return locations; + } catch (Exception ex) { + // No return + ServletOps.errorBadRequest("TDB database location can not be deterined"); + return null; + } } }
jena-fuseki2/jena-fuseki-webapp/src/main/java/org/apache/jena/fuseki/webapp/FusekiWebapp.java+26 −24 modified@@ -18,8 +18,6 @@ package org.apache.jena.fuseki.webapp; -import static java.lang.String.format; - import java.io.File; import java.io.IOException; import java.io.InputStream; @@ -39,12 +37,12 @@ import org.apache.jena.fuseki.FusekiConfigException; import org.apache.jena.fuseki.build.FusekiConfig; import org.apache.jena.fuseki.cmd.FusekiArgs; +import org.apache.jena.fuseki.mgt.ActionDatasets; import org.apache.jena.fuseki.mgt.Template; import org.apache.jena.fuseki.mgt.TemplateFunctions; import org.apache.jena.fuseki.server.DataAccessPoint; import org.apache.jena.fuseki.server.DataAccessPointRegistry; import org.apache.jena.fuseki.server.DataService; -import org.apache.jena.fuseki.servlets.HttpAction; import org.apache.jena.fuseki.servlets.ServletOps; import org.apache.jena.fuseki.system.FusekiCore; import org.apache.jena.graph.Graph; @@ -115,6 +113,27 @@ public class FusekiWebapp // Marks the end of successful initialization. /*package*/static boolean serverInitialized = false; + // Run-time lock for operations that change the server configuration (e..g adding and deleting data services) + public static final Object systemLock = new Object(); + + /** + * Control whether to allow creating new dataservices by uploading a config file. + * See {@link ActionDatasets}. + * + */ + public static final String allowConfigFileProperty = "fuseki:allowAddByConfigFile"; + + /** + * Return whether to allow service configuration files to be uploaded as a file. + * See {@link ActionDatasets}. + */ + public static boolean allowConfigFiles() { + String value = System.getProperty(allowConfigFileProperty); + if ( value != null ) + return "true".equals(value); + return false; + } + public /*package*/ synchronized static void formatBaseArea() { if ( initialized ) return; @@ -407,25 +426,6 @@ private static Path makePath(Path root , String relName ) { return path; } - /** - * Dataset set name to configuration file name. Return a configuration file name - - * existing one or ".ttl" form if new - */ - public static String datasetNameToConfigurationFile(HttpAction action, String dsName) { - List<String> existing = existingConfigurationFile(dsName); - if ( ! existing.isEmpty() ) { - if ( existing.size() > 1 ) { - action.log.warn(format("[%d] Multiple existing configuration files for %s : %s", - action.id, dsName, existing)); - ServletOps.errorBadRequest("Multiple existing configuration files for "+dsName); - return null; - } - return existing.get(0).toString(); - } - - return generateConfigurationFilename(dsName); - } - /** New configuration file name - absolute filename */ public static String generateConfigurationFilename(String dsName) { String filename = dsName; @@ -437,10 +437,12 @@ public static String generateConfigurationFilename(String dsName) { } /** Return the filenames of all matching files in the configuration directory (absolute paths returned ). */ - public static List<String> existingConfigurationFile(String baseFilename) { + public static List<String> existingConfigurationFile(String serviceName) { + String filename = DataAccessPoint.isCanonical(serviceName) ? serviceName.substring(1) : serviceName; try { List<String> paths = new ArrayList<>(); - try (DirectoryStream<Path> stream = Files.newDirectoryStream(FusekiWebapp.dirConfiguration, baseFilename+".*") ) { + // This ".* is a file glob pattern, not a regular expression - it looks for file extensions. + try (DirectoryStream<Path> stream = Files.newDirectoryStream(FusekiWebapp.dirConfiguration, filename+".*") ) { stream.forEach((p)-> paths.add(FusekiWebapp.dirConfiguration.resolve(p).toString() )); } return paths;
jena-fuseki2/jena-fuseki-webapp/src/test/java/org/apache/jena/fuseki/TestWebappAdminAddDeleteDatasetFile.java+679 −0 added@@ -0,0 +1,679 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.jena.fuseki; + +import static org.apache.jena.fuseki.mgt.ServerMgtConst.opBackup; +import static org.apache.jena.fuseki.mgt.ServerMgtConst.opCompact; +import static org.apache.jena.fuseki.mgt.ServerMgtConst.opDatasets; +import static org.apache.jena.fuseki.mgt.ServerMgtConst.opListBackups; +import static org.apache.jena.fuseki.server.ServerConst.opStats; +import static org.apache.jena.http.HttpOp.*; +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertNotNull; +import static org.junit.Assert.assertThrows; +import static org.junit.Assert.assertTrue; + +import java.io.FileNotFoundException; +import java.io.IOException; +import java.net.http.HttpRequest.BodyPublisher; +import java.net.http.HttpRequest.BodyPublishers; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.Paths; +import java.util.ArrayList; +import java.util.List; +import java.util.Objects; +import java.util.concurrent.TimeUnit; + +import org.apache.commons.lang3.SystemUtils; +import org.apache.jena.atlas.io.IO; +import org.apache.jena.atlas.json.JSON; +import org.apache.jena.atlas.json.JsonArray; +import org.apache.jena.atlas.json.JsonObject; +import org.apache.jena.atlas.json.JsonValue; +import org.apache.jena.atlas.junit.AssertExtra; +import org.apache.jena.atlas.lib.Lib; +import org.apache.jena.atlas.logging.LogCtl; +import org.apache.jena.atlas.web.HttpException; +import org.apache.jena.atlas.web.TypedInputStream; +import org.apache.jena.fuseki.ctl.JsonConstCtl; +import org.apache.jena.fuseki.test.HttpTest; +import org.apache.jena.fuseki.webapp.FusekiWebapp; +import org.apache.jena.query.QueryExecution; +import org.apache.jena.rdfconnection.RDFConnection; +import org.apache.jena.riot.WebContent; +import org.apache.jena.web.HttpSC; +import org.awaitility.Awaitility; +import org.junit.After; +import org.junit.Assert; +import org.junit.AssumptionViolatedException; +import org.junit.Before; +import org.junit.jupiter.api.Test; + +/** Tests of the admin functionality */ +public class TestWebappAdminAddDeleteDatasetFile extends AbstractFusekiWebappTest { + + // Name of the dataset in the assembler file. + static String dsTest = "test-ds1"; + static String dsTestInf = "test-ds4"; + + // There are two Fuseki-TDB2 tests: add_delete_dataset_6() and compact_01(). + // + // On certain build systems (GH action/Linux under load, ASF Jenkins sometimes), + // add_delete_dataset_6 fails (transactions active), or compact_01 (gets a 404), + // if the two databases are the same. + static String dsTestTdb2a = "test-tdb2a"; + static String dsTestTdb2b = "test-tdb2b"; + static String fileBase = "testing/"; + + @Before public void setLogging() { + LogCtl.setLevel(Fuseki.backupLogName, "ERROR"); + LogCtl.setLevel(Fuseki.compactLogName,"ERROR"); + LogCtl.setLevel(Fuseki.adminLogName,"ERROR"); + Awaitility.setDefaultPollDelay(20,TimeUnit.MILLISECONDS); + Awaitility.setDefaultPollInterval(50,TimeUnit.MILLISECONDS); + } + + @After public void unsetLogging() { + LogCtl.setLevel(Fuseki.backupLogName, "WARN"); + LogCtl.setLevel(Fuseki.compactLogName,"WARN"); + LogCtl.setLevel(Fuseki.adminLogName,"WARN"); + } + + private static void withFileEnabled(Runnable action) { + System.setProperty(FusekiWebapp.allowConfigFileProperty, "true"); + try { + action.run(); + } finally { + System.getProperties().remove(FusekiWebapp.allowConfigFileProperty); + } + } + + // --- List all datasets + + @Test public void list_datasets_1() { + try ( TypedInputStream in = httpGet(ServerCtl.urlRoot()+"$/"+opDatasets); ) { + IO.skipToEnd(in); + } + } + + @Test public void list_datasets_2() { + try ( TypedInputStream in = httpGet(ServerCtl.urlRoot()+"$/"+opDatasets) ) { + assertEqualsIgnoreCase(WebContent.contentTypeJSON, in.getContentType()); + JsonValue v = JSON.parseAny(in); + assertNotNull(v.getAsObject().get("datasets")); + checkJsonDatasetsAll(v); + } + } + + // Specific dataset + @Test public void list_datasets_3() { + checkExists(ServerCtl.datasetName()); + } + + // Specific dataset + @Test public void list_datasets_4() { + HttpTest.expect404( () -> getDatasetDescription("does-not-exist") ); + } + + // Specific dataset + @Test public void list_datasets_5() { + JsonValue v = getDatasetDescription(ServerCtl.datasetName()); + checkJsonDatasetsOne(v.getAsObject()); + } + + @Test public void add_dataset_blocked() { + // Do without enabling upload of configuration files - expected to fail + HttpException ex = assertThrows(HttpException.class, ()->addTestDatasetPerform(fileBase+"config-ds-plain-1.ttl")); + assertEquals(ex.getStatusCode(), HttpSC.BAD_REQUEST_400); + } + + // Specific dataset + @Test public void add_delete_dataset_1() { + checkNotThere(dsTest); + + addTestDataset(); + + // Check exists. + checkExists(dsTest); + + // Remove it. + deleteDataset(dsTest); + checkNotThere(dsTest); + } + + // Try to add twice + @Test public void add_delete_dataset_2() { + checkNotThere(dsTest); + + withFileEnabled(()->{ + try { + Path f = Path.of(fileBase+"config-ds-plain-1.ttl"); + { + httpPost(ServerCtl.urlRoot()+"$/"+opDatasets, + WebContent.contentTypeTurtle+"; charset="+WebContent.charsetUTF8, + BodyPublishers.ofFile(f)); + } + // Check exists. + checkExists(dsTest); + try { + } catch (HttpException ex) { + httpPost(ServerCtl.urlRoot()+"$/"+opDatasets, + WebContent.contentTypeTurtle+"; charset="+WebContent.charsetUTF8, + BodyPublishers.ofFile(f)); + assertEquals(HttpSC.CONFLICT_409, ex.getStatusCode()); + } + } catch (IOException ex) { IO.exception(ex); return; } + }); + // Check exists. + checkExists(dsTest); + deleteDataset(dsTest); + } + + @Test public void add_delete_dataset_3() { + checkNotThere(dsTest); + addTestDataset(); + checkExists(dsTest); + deleteDataset(dsTest); + checkNotThere(dsTest); + addTestDataset(); + checkExists(dsTest); + deleteDataset(dsTest); + } + + @Test public void add_delete_dataset_4() { + checkNotThere(dsTest); + checkNotThere(dsTestInf); + addTestDatasetInf(); + checkNotThere(dsTest); + checkExists(dsTestInf); + + deleteDataset(dsTestInf); + checkNotThere(dsTestInf); + addTestDatasetInf(); + checkExists(dsTestInf); + deleteDataset(dsTestInf); + } + + @Test public void add_delete_dataset_5() { + // New style operations : cause two fuseki:names + addTestDataset(fileBase+"config-ds-plain-2.ttl"); + checkExists("test-ds2"); + } + + @Test public void add_delete_dataset_6() { + String testDB = dsTestTdb2a; + assumeNotWindows(); + + checkNotThere(testDB); + + addTestDatasetTDB2(testDB); + + // Check exists. + checkExists(testDB); + + // Remove it. + deleteDataset(testDB); + checkNotThere(testDB); + } + + @Test public void add_delete_dataset_TDB_1() { + String testDB = dsTestTdb2a; + assumeNotWindows(); + + checkNotThere(testDB); + + addTestDatasetTDB2(testDB); + + // Check exists. + checkExists(testDB); + + // Remove it. + deleteDataset(testDB); + checkNotThere(testDB); + } + + @Test public void add_delete_dataset_TDB_2() { + // This has location "--mem--" + String testDB = dsTestTdb2b; + checkNotThere(testDB); + addTestDatasetTDB2(testDB); + // Check exists. + checkExists(testDB); + // Remove it. + deleteDataset(testDB); + checkNotThere(testDB); + } + + @Test public void add_error_1() { + HttpTest.execWithHttpException(HttpSC.BAD_REQUEST_400, + ()-> addTestDataset(fileBase+"config-ds-bad-name-1.ttl")); + } + + @Test public void add_error_2() { + HttpTest.execWithHttpException(HttpSC.BAD_REQUEST_400, + ()-> addTestDataset(fileBase+"config-ds-bad-name-2.ttl")); + } + + @Test public void add_error_3() { + HttpTest.execWithHttpException(HttpSC.BAD_REQUEST_400, + ()-> addTestDataset(fileBase+"config-ds-bad-name-3.ttl")); + } + + @Test public void add_error_4() { + HttpTest.execWithHttpException(HttpSC.BAD_REQUEST_400, + ()-> addTestDataset(fileBase+"config-ds-bad-name-4.ttl")); + } + + @Test public void noOverwriteExistingConfigFile() throws IOException { + var workingDir = Paths.get("").toAbsolutePath(); + var path = workingDir.resolve(TS_FusekiWebapp.FusekiTestBase+"/configuration/test-ds0-empty.ttl"); + var dbConfig = path.toFile(); + dbConfig.createNewFile(); + try { + // refresh the file system so that the file exists + dbConfig = path.toFile(); + assertTrue (dbConfig.exists()); + assertEquals(0, dbConfig.length()); + + // Try to override the file with a new configuration. + String ct = WebContent.contentTypeHTMLForm; + String body = "dbName=test-ds0-empty&dbType=mem"; + HttpException ex = assertThrows(org.apache.jena.atlas.web.HttpException.class, + ()-> httpPost(ServerCtl.urlRoot()+"$/" + opDatasets, ct, body)); + assertEquals(HttpSC.CONFLICT_409, ex.getStatusCode()); + // refresh the file system + dbConfig = path.toFile(); + assertTrue(dbConfig.exists()); + assertEquals("File should be still empty", 0, dbConfig.length()); + } + finally { + // Clean up the file. + if (Files.exists(path)) { + Files.delete(path); + } + } + } + + @Test public void delete_dataset_1() { + String name = "NoSuchDataset"; + HttpTest.expect404( ()-> httpDelete(ServerCtl.urlRoot()+"$/"+opDatasets+"/"+name) ); + } + + // ---- Backup + + @Test public void create_backup_1() { + String id = null; + try { + JsonValue v = httpPostRtnJSON(ServerCtl.urlRoot() + "$/" + opBackup + "/" + ServerCtl.datasetName()); + id = v.getAsObject().getString("taskId"); + } finally { + waitForTasksToFinish(1000, 10, 20000); + } + Assert.assertNotNull(id); + checkInTasks(id); + + // Check a backup was created + try ( TypedInputStream in = httpGet(ServerCtl.urlRoot()+"$/"+opListBackups) ) { + assertEqualsIgnoreCase(WebContent.contentTypeJSON, in.getContentType()); + JsonValue v = JSON.parseAny(in); + assertNotNull(v.getAsObject().get("backups")); + JsonArray a = v.getAsObject().get("backups").getAsArray(); + Assert.assertEquals(1, a.size()); + } + + JsonValue task = getTask(id); + Assert.assertNotNull(id); + // Expect task success + Assert.assertTrue("Expected task to be marked as successful", task.getAsObject().getBoolean(JsonConstCtl.success)); + } + + @Test + public void create_backup_2() { + HttpTest.expect400(()->{ + JsonValue v = httpPostRtnJSON(ServerCtl.urlRoot() + "$/" + opBackup + "/noSuchDataset"); + }); + } + + @Test public void list_backups_1() { + try ( TypedInputStream in = httpGet(ServerCtl.urlRoot()+"$/"+opListBackups) ) { + assertEqualsIgnoreCase(WebContent.contentTypeJSON, in.getContentType()); + JsonValue v = JSON.parseAny(in); + assertNotNull(v.getAsObject().get("backups")); + } + } + + // ---- Compact + + @Test public void compact_01() { + assumeNotWindows(); + + String testDB = dsTestTdb2b; + try { + checkNotThere(testDB); + addTestDatasetTDB2(testDB); + checkExists(testDB); + + String id = null; + try { + JsonValue v = httpPostRtnJSON(ServerCtl.urlRoot() + "$/" + opCompact + "/" + testDB); + id = v.getAsObject().getString(JsonConstCtl.taskId); + } finally { + waitForTasksToFinish(1000, 500, 20_000); + } + Assert.assertNotNull(id); + checkInTasks(id); + + JsonValue task = getTask(id); + // ---- + // The result assertion is throwing NPE occasionally on some heavily loaded CI servers. + // This may be because of server or test code encountering a very long wait. + // These next statements check the assumed structure of the return. + Assert.assertNotNull("Task value", task); + JsonObject obj = task.getAsObject(); + Assert.assertNotNull("Task.getAsObject()", obj); + // Provoke code to get a stacktrace. + obj.getBoolean(JsonConstCtl.success); + // ---- + // The assertion we really wanted to check. + // Check task success + Assert.assertTrue("Expected task to be marked as successful", task.getAsObject().getBoolean(JsonConstCtl.success)); + } finally { + deleteDataset(testDB); + } + } + + @Test public void compact_02() { + HttpTest.expect400(()->{ + JsonValue v = httpPostRtnJSON(ServerCtl.urlRoot() + "$/" + opCompact + "/noSuchDataset"); + }); + } + + private void assumeNotWindows() { + if (SystemUtils.IS_OS_WINDOWS) + throw new AssumptionViolatedException("Test may be unstable on Windows due to inability to delete memory-mapped files"); + } + + @Test public void stats_1() { + JsonValue v = execGetJSON(ServerCtl.urlRoot()+"$/"+opStats); + checkJsonStatsAll(v); + } + + @Test public void stats_2() { + addTestDataset(); + JsonValue v = execGetJSON(ServerCtl.urlRoot()+"$/"+opStats+ServerCtl.datasetPath()); + checkJsonStatsAll(v); + deleteDataset(dsTest); + } + + @Test public void stats_3() { + addTestDataset(); + HttpTest.expect404(()-> execGetJSON(ServerCtl.urlRoot()+"$/"+opStats+"/DoesNotExist")); + deleteDataset(dsTest); + } + + @Test public void stats_4() { + JsonValue v = execPostJSON(ServerCtl.urlRoot()+"$/"+opStats); + checkJsonStatsAll(v); + } + + @Test public void stats_5() { + addTestDataset(); + JsonValue v = execPostJSON(ServerCtl.urlRoot()+"$/"+opStats+ServerCtl.datasetPath()); + checkJsonStatsAll(v); + deleteDataset(dsTest); + } + + // Async task testing + + private void assertEqualsIgnoreCase(String contenttypejson, String contentType) {} + + private static JsonValue getTask(String taskId) { + String url = ServerCtl.urlRoot()+"$/tasks/"+taskId; + return httpGetJson(url); + } + + private static JsonValue getDatasetDescription(String dsName) { + if ( dsName.startsWith("/") ) + dsName = dsName.substring(1); + try (TypedInputStream in = httpGet(ServerCtl.urlRoot() + "$/" + opDatasets + "/" + dsName)) { + AssertExtra.assertEqualsIgnoreCase(WebContent.contentTypeJSON, in.getContentType()); + JsonValue v = JSON.parse(in); + return v; + } + } + + // -- Add + + private static void addTestDataset() { + addTestDataset(fileBase+"config-ds-plain-1.ttl"); + } + + private static void addTestDatasetInf() { + addTestDataset(fileBase+"config-ds-inf.ttl"); + } + + private static void addTestDatasetTDB2(String DBname) { + Objects.nonNull(DBname); + if ( DBname.equals(dsTestTdb2a) ) { + addTestDataset(fileBase+"config-tdb2a.ttl"); + return; + } + if ( DBname.equals(dsTestTdb2b) ) { + addTestDataset(fileBase+"config-tdb2b.ttl"); + return; + } + throw new IllegalArgumentException("No configuration for "+DBname); + } + + private static void addTestDataset(String filename) { + withFileEnabled(()->{ + addTestDatasetPerform(filename); + }); + } + + private static void addTestDatasetPerform(String filename) { + try { + Path f = Path.of(filename); + BodyPublisher body = BodyPublishers.ofFile(f); + String ct = WebContent.contentTypeTurtle; + httpPost(ServerCtl.urlRoot()+"$/"+opDatasets, ct, body); + } catch (FileNotFoundException e) { + IO.exception(e); + } + } + + private static void deleteDataset(String name) { + httpDelete(ServerCtl.urlRoot()+"$/"+opDatasets+"/"+name); + } + + private static void checkTask(JsonValue v) { + assertNotNull(v); + assertTrue(v.isObject()); + //System.out.println(v); + JsonObject obj = v.getAsObject(); + try { + assertTrue(obj.hasKey("task")); + assertTrue(obj.hasKey("taskId")); + // Not present until it runs : "started" + } catch (AssertionError ex) { + System.out.println(obj); + throw ex; + } + } + + private static void checkInTasks(String x) { + String url = ServerCtl.urlRoot()+"$/tasks"; + JsonValue v = httpGetJson(url); + assertTrue(v.isArray()); + JsonArray array = v.getAsArray(); + int found = 0; + for ( int i = 0; i < array.size(); i++ ) { + JsonValue jv = array.get(i); + assertTrue(jv.isObject()); + JsonObject obj = jv.getAsObject(); + checkTask(obj); + if ( obj.getString("taskId").equals(x) ) { + found++; + } + } + assertEquals("Occurrence of taskId count", 1, found); + } + + private static List<String> runningTasks(String... x) { + String url = ServerCtl.urlRoot()+"$/tasks"; + JsonValue v = httpGetJson(url); + assertTrue(v.isArray()); + JsonArray array = v.getAsArray(); + List<String> running = new ArrayList<>(); + for ( int i = 0; i < array.size(); i++ ) { + JsonValue jv = array.get(i); + assertTrue(jv.isObject()); + JsonObject obj = jv.getAsObject(); + if ( isRunning(obj) ) + running.add(obj.getString("taskId")); + } + return running; + } + + /** + * Wait for tasks to all finish. + * Algorithm: wait for {@code pause}, then start polling for upto {@code maxWaitMillis}. + * Intervals in milliseconds. + * @param pauseMillis + * @param pollInterval + * @param maxWaitMillis + * @return + */ + private static boolean waitForTasksToFinish(int pauseMillis, int pollInterval, int maxWaitMillis) { + // Wait for them to finish. + // Divide into chunks + if ( pauseMillis > 0 ) + Lib.sleep(pauseMillis); + long start = System.currentTimeMillis(); + long endTime = start + maxWaitMillis; + final int intervals = maxWaitMillis/pollInterval; + long now = start; + for (int i = 0 ; i < intervals ; i++ ) { + // May have waited (much) longer than the pollInterval : heavily loaded build systems. + if ( now-start > maxWaitMillis ) + break; + List<String> x = runningTasks(); + if ( x.isEmpty() ) + return true; + Lib.sleep(pollInterval); + now = System.currentTimeMillis(); + } + return false; + } + + private static boolean isRunning(JsonObject taskObj) { + checkTask(taskObj); + return taskObj.hasKey("started") && ! taskObj.hasKey("finished"); + } + + private static void askPing(String name) { + if ( name.startsWith("/") ) + name = name.substring(1); + try ( TypedInputStream in = httpGet(ServerCtl.urlRoot()+name+"/sparql?query=ASK%7B%7D") ) { + IO.skipToEnd(in); + } + } + + private static void adminPing(String name) { + try ( TypedInputStream in = httpGet(ServerCtl.urlRoot()+"$/"+opDatasets+"/"+name) ) { + IO.skipToEnd(in); + } + } + + private static void checkExists(String name) { + adminPing(name); + askPing(name); + } + + private static void checkNotThere(String name) { + String n = (name.startsWith("/")) ? name.substring(1) : name; + // Check gone exists. + HttpTest.expect404(()-> adminPing(n) ); + HttpTest.expect404(() -> askPing(n) ); + } + + private static void checkJsonDatasetsAll(JsonValue v) { + assertNotNull(v.getAsObject().get("datasets")); + JsonArray a = v.getAsObject().get("datasets").getAsArray(); + for ( JsonValue v2 : a ) + checkJsonDatasetsOne(v2); + } + + private static void checkJsonDatasetsOne(JsonValue v) { + assertTrue(v.isObject()); + JsonObject obj = v.getAsObject(); + assertNotNull(obj.get("ds.name")); + assertNotNull(obj.get("ds.services")); + assertNotNull(obj.get("ds.state")); + assertTrue(obj.get("ds.services").isArray()); + } + + private static void checkJsonStatsAll(JsonValue v) { + assertNotNull(v.getAsObject().get("datasets")); + JsonObject a = v.getAsObject().get("datasets").getAsObject(); + for ( String dsname : a.keys() ) { + JsonValue obj = a.get(dsname).getAsObject(); + checkJsonStatsOne(obj); + } + } + + private static void checkJsonStatsOne(JsonValue v) { + checkJsonStatsCounters(v); + JsonObject obj1 = v.getAsObject().get("endpoints").getAsObject(); + for ( String srvName : obj1.keys() ) { + JsonObject obj2 = obj1.get(srvName).getAsObject(); + assertTrue(obj2.hasKey("description")); + assertTrue(obj2.hasKey("operation")); + checkJsonStatsCounters(obj2); + } + } + + private static void checkJsonStatsCounters(JsonValue v) { + JsonObject obj = v.getAsObject(); + assertTrue(obj.hasKey("Requests")); + assertTrue(obj.hasKey("RequestsGood")); + assertTrue(obj.hasKey("RequestsBad")); + } + + private static JsonValue execGetJSON(String url) { + try ( TypedInputStream in = httpGet(url) ) { + AssertExtra.assertEqualsIgnoreCase(WebContent.contentTypeJSON, in.getContentType()); + return JSON.parse(in); + } + } + + private static JsonValue execPostJSON(String url) { + try ( TypedInputStream in = httpPostStream(url, null, null, null) ) { + AssertExtra.assertEqualsIgnoreCase(WebContent.contentTypeJSON, in.getContentType()); + return JSON.parse(in); + } + } + + static int count(RDFConnection conn) { + try ( QueryExecution qExec = conn.query("SELECT (count(*) AS ?C) { ?s ?p ?o }")) { + return qExec.execSelect().next().getLiteral("C").getInt(); + } + } +}
jena-fuseki2/jena-fuseki-webapp/src/test/java/org/apache/jena/fuseki/TestWebappAdminAddDeleteDatasetTemplate.java+268 −0 added@@ -0,0 +1,268 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.jena.fuseki; + +import static org.apache.jena.fuseki.mgt.ServerMgtConst.opDatasets; +import static org.apache.jena.http.HttpOp.httpPost; +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertTrue; +import static org.junit.Assert.fail; + +import java.io.FileNotFoundException; +import java.net.http.HttpRequest.BodyPublisher; +import java.net.http.HttpRequest.BodyPublishers; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.concurrent.TimeUnit; + +import org.apache.jena.atlas.io.IO; +import org.apache.jena.atlas.logging.LogCtl; +import org.apache.jena.atlas.web.HttpException; +import org.apache.jena.atlas.web.TypedInputStream; +import org.apache.jena.base.Sys; +import org.apache.jena.fuseki.webapp.FusekiWebapp; +import org.apache.jena.http.HttpOp; +import org.apache.jena.query.QueryExecution; +import org.apache.jena.rdfconnection.RDFConnection; +import org.apache.jena.riot.WebContent; +import org.apache.jena.sparql.exec.http.Params; +import org.apache.jena.web.HttpSC; +import org.apache.jena.web.HttpSC.Code; +import org.awaitility.Awaitility; +import org.junit.After; +import org.junit.Before; +import org.junit.Test; + +/** Tests of the admin functionality */ +public class TestWebappAdminAddDeleteDatasetTemplate extends AbstractFusekiWebappTest { + + // Name of the dataset in the assembler file. + static String dsTest = "test-ds1"; + static String dsTestInf = "test-ds4"; + + // There are two Fuseki-TDB2 tests: add_delete_dataset_6() and compact_01(). + // + // On certain build systems (GH action/Linux under load, ASF Jenkins sometimes), + // add_delete_dataset_6 fails (transactions active), or compact_01 (gets a 404), + // if the two databases are the same. + static String dsTestTdb2a = "test-tdb2a"; + static String dsTestTdb2b = "test-tdb2b"; + static String fileBase = "testing/"; + + @Before public void setLogging() { + LogCtl.setLevel(Fuseki.backupLogName, "ERROR"); + LogCtl.setLevel(Fuseki.compactLogName,"ERROR"); + LogCtl.setLevel(Fuseki.adminLogName,"ERROR"); + Awaitility.setDefaultPollDelay(20,TimeUnit.MILLISECONDS); + Awaitility.setDefaultPollInterval(50,TimeUnit.MILLISECONDS); + } + + @After public void unsetLogging() { + LogCtl.setLevel(Fuseki.backupLogName, "WARN"); + LogCtl.setLevel(Fuseki.compactLogName,"WARN"); + LogCtl.setLevel(Fuseki.adminLogName,"WARN"); + } + + @Test public void add_dataset_01() { + testAddDataset("db_1"); + } + + @Test public void add_dataset_02() { + testAddDataset( "/db_2"); + } + + // Do as a file - which is blocked. + @Test public void add_dataset_99() { + expect400(()->addTestDataset(fileBase+"config-ds-plain-1.ttl")); + } + + private static void addTestDataset(String filename) { + try { + Path f = Path.of(filename); + BodyPublisher body = BodyPublishers.ofFile(f); + String ct = WebContent.contentTypeTurtle; + httpPost(ServerCtl.urlRoot()+"$/"+opDatasets, ct, body); + } catch (FileNotFoundException e) { + IO.exception(e); + } + } + + @Test public void add_dataset_bad_02() { + badAddDataserverRequest("bad_2 illegal"); + } + + @Test public void add_dataset_bad_03() { + badAddDataserverRequest("bad_3/path"); + } + + @Test public void add_dataset_bad_04() { + badAddDataserverRequest(""); + } + + @Test public void add_dataset_bad_05() { + badAddDataserverRequest(" "); + } + + @Test public void add_dataset_bad_06() { + badAddDataserverRequest("bad_6_AB CD"); + } + + @Test public void add_dataset_bad_07() { + badAddDataserverRequest(".."); + } + + @Test public void add_dataset_bad_08() { + badAddDataserverRequest("/.."); + } + + @Test public void add_dataset_bad_09() { + badAddDataserverRequest("/../elsewhere"); + } + + @Test public void add_dataset_bad_10() { + badAddDataserverRequest("//bad_10"); + } + + // add-delete + + @Test public void add_delete_mem_1() { + testAddDeleteAdd("db_add_delete_1", "mem", false, false); + } + + @Test public void add_delete_tdb_1() { + if ( Sys.isWindows ) + return; + testAddDeleteAdd("db_add_delete_tdb_1", "tdb2", false, true); + } + + @Test public void add_delete_tdb_2() { + if ( Sys.isWindows ) + return; + String dbName = "db_add_delete_tdb_2"; + testAddDeleteAdd(dbName, "tdb2", false, true); + } + + // Attempt to add a in-memory dataset. Used to test the name checking. + private void testAddDataset(String dbName) { + Params params = Params.create().add("dbName", dbName).add("dbType", "mem"); + // Use the template + String actionURL = ServerCtl.urlRoot()+"$/datasets"; + HttpOp.httpPostForm(actionURL, params); + String datasetURL = dbName.startsWith("/") + ? ServerCtl.urlRoot()+(dbName.substring(1)) + : ServerCtl.urlRoot()+dbName; + assertTrue(exists(datasetURL)); + } + + private void testAddDeleteAdd(String dbName, String dbType, boolean alreadyExists, boolean hasFiles) { + String datasetURL = ServerCtl.urlRoot()+dbName; + Params params = Params.create().add("dbName", dbName).add("dbType", dbType); + + if ( alreadyExists ) + assertTrue(exists(datasetURL)); + else + assertFalse(exists(datasetURL)); + + // Use the template + HttpOp.httpPostForm(ServerCtl.urlRoot()+"$/datasets", params); + + RDFConnection conn = RDFConnection.connect(ServerCtl.urlRoot()+dbName); + conn.update("INSERT DATA { <x:s> <x:p> 123 }"); + int x1 = count(conn); + assertEquals(1, x1); + + Path pathDB = FusekiWebapp.dirDatabases.resolve(dbName); + + if ( hasFiles ) + assertTrue(Files.exists(pathDB)); + + HttpOp.httpDelete(ServerCtl.urlRoot()+"$/datasets/"+dbName); + + assertFalse(exists(datasetURL)); + + //if ( hasFiles ) + assertFalse(Files.exists(pathDB)); + + // Recreate : no contents. + HttpOp.httpPostForm(ServerCtl.urlRoot()+"$/datasets", params); + assertTrue(exists(datasetURL)); + int x2 = count(conn); + assertEquals(0, x2); + if ( hasFiles ) + assertTrue(Files.exists(pathDB)); + } + + private void badAddDataserverRequest(String dbName) { + expect400(()->testAddDataset(dbName)); + } + + private static boolean exists(String url) { + try ( TypedInputStream in = HttpOp.httpGet(url) ) { + return true; + } catch (HttpException ex) { + if ( ex.getStatusCode() == HttpSC.NOT_FOUND_404 ) + return false; + throw ex; + } + } + + static int count(RDFConnection conn) { + try ( QueryExecution qExec = conn.query("SELECT (count(*) AS ?C) { ?s ?p ?o }")) { + return qExec.execSelect().next().getLiteral("C").getInt(); + } + } + + // -- From fusekiTestLib + + public static void expect400(Runnable runnable) { + expectFail(runnable, HttpSC.Code.BAD_REQUEST); + } + + public static void expect401(Runnable runnable) { + expectFail(runnable, HttpSC.Code.UNAUTHORIZED); + } + + public static void expect403(Runnable runnable) { + expectFail(runnable, HttpSC.Code.FORBIDDEN); + } + + public static void expect404(Runnable runnable) { + expectFail(runnable, HttpSC.Code.NOT_FOUND); + } + + public static void expect409(Runnable runnable) { + expectFail(runnable, HttpSC.Code.CONFLICT); + } + + public static void expectFail(Runnable runnable, Code code) { + if ( code == null || ( 200 <= code.getCode() && code.getCode() < 300 ) ) { + runnable.run(); + return; + } + try { + runnable.run(); + fail("Failed: Got no exception: Expected HttpException "+code.getCode()); + } catch (HttpException ex) { + if ( ex.getStatusCode() == code.getCode() ) + return; + throw ex; + } + } +}
jena-fuseki2/jena-fuseki-webapp/src/test/java/org/apache/jena/fuseki/TestWebappAdmin.java+16 −365 modified@@ -18,24 +18,25 @@ package org.apache.jena.fuseki; -import static org.apache.jena.fuseki.mgt.ServerMgtConst.*; +import static org.apache.jena.fuseki.mgt.ServerMgtConst.opBackup; +import static org.apache.jena.fuseki.mgt.ServerMgtConst.opDatasets; +import static org.apache.jena.fuseki.mgt.ServerMgtConst.opListBackups; +import static org.apache.jena.fuseki.mgt.ServerMgtConst.opServer; import static org.apache.jena.fuseki.server.ServerConst.opPing; -import static org.apache.jena.fuseki.server.ServerConst.opStats; -import static org.apache.jena.http.HttpOp.*; +import static org.apache.jena.http.HttpOp.httpGet; +import static org.apache.jena.http.HttpOp.httpGetJson; +import static org.apache.jena.http.HttpOp.httpPost; +import static org.apache.jena.http.HttpOp.httpPostRtnJSON; import static org.awaitility.Awaitility.await; -import static org.junit.Assert.*; +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertNotNull; +import static org.junit.Assert.assertTrue; +import static org.junit.Assert.fail; -import java.io.FileNotFoundException; -import java.io.IOException; -import java.net.http.HttpRequest.BodyPublisher; -import java.net.http.HttpRequest.BodyPublishers; -import java.nio.file.Path; import java.util.ArrayList; import java.util.List; -import java.util.Objects; import java.util.concurrent.TimeUnit; -import org.apache.commons.lang3.SystemUtils; import org.apache.jena.atlas.io.IO; import org.apache.jena.atlas.json.JSON; import org.apache.jena.atlas.json.JsonArray; @@ -53,22 +54,15 @@ import org.apache.jena.riot.WebContent; import org.apache.jena.web.HttpSC; import org.awaitility.Awaitility; -import org.junit.*; +import org.junit.After; +import org.junit.Assert; +import org.junit.Before; +import org.junit.Test; /** Tests of the admin functionality */ public class TestWebappAdmin extends AbstractFusekiWebappTest { // Name of the dataset in the assembler file. - static String dsTest = "test-ds1"; - static String dsTestInf = "test-ds4"; - - // There are two Fuseki-TDB2 tests: add_delete_dataset_6() and compact_01(). - // - // On certain build systems (GH action/Linux under load, ASF Jenkins sometimes), - // add_delete_dataset_6 fails (transactions active), or compact_01 (gets a 404), - // if the two databases are the same. - static String dsTestTdb2a = "test-tdb2a"; - static String dsTestTdb2b = "test-tdb2b"; static String fileBase = "testing/"; @Before public void setLogging() { @@ -141,152 +135,6 @@ public class TestWebappAdmin extends AbstractFusekiWebappTest { checkJsonDatasetsOne(v.getAsObject()); } - // Specific dataset - @Test public void add_delete_dataset_1() { - checkNotThere(dsTest); - - addTestDataset(); - - // Check exists. - checkExists(dsTest); - - // Remove it. - deleteDataset(dsTest); - checkNotThere(dsTest); - } - - // Try to add twice - @Test public void add_delete_dataset_2() { - checkNotThere(dsTest); - - try { - Path f = Path.of(fileBase+"config-ds-plain-1.ttl"); - { - httpPost(ServerCtl.urlRoot()+"$/"+opDatasets, - WebContent.contentTypeTurtle+"; charset="+WebContent.charsetUTF8, - BodyPublishers.ofFile(f)); - } - // Check exists. - checkExists(dsTest); - try { - } catch (HttpException ex) { - httpPost(ServerCtl.urlRoot()+"$/"+opDatasets, - WebContent.contentTypeTurtle+"; charset="+WebContent.charsetUTF8, - BodyPublishers.ofFile(f)); - assertEquals(HttpSC.CONFLICT_409, ex.getStatusCode()); - } - } catch (IOException ex) { IO.exception(ex); return; } - - // Check exists. - checkExists(dsTest); - deleteDataset(dsTest); - } - - @Test public void add_delete_dataset_3() { - checkNotThere(dsTest); - addTestDataset(); - checkExists(dsTest); - deleteDataset(dsTest); - checkNotThere(dsTest); - addTestDataset(); - checkExists(dsTest); - deleteDataset(dsTest); - } - - @Test public void add_delete_dataset_4() { - checkNotThere(dsTest); - checkNotThere(dsTestInf); - addTestDatasetInf(); - checkNotThere(dsTest); - checkExists(dsTestInf); - - deleteDataset(dsTestInf); - checkNotThere(dsTestInf); - addTestDatasetInf(); - checkExists(dsTestInf); - deleteDataset(dsTestInf); - } - - @Test public void add_delete_dataset_5() { - // New style operations : cause two fuseki:names - addTestDataset(fileBase+"config-ds-plain-2.ttl"); - checkExists("test-ds2"); - } - - @Test public void add_delete_dataset_6() { - String testDB = dsTestTdb2a; - assumeNotWindows(); - - checkNotThere(testDB); - - addTestDatasetTDB2(testDB); - - // Check exists. - checkExists(testDB); - - // Remove it. - deleteDataset(testDB); - checkNotThere(testDB); - } - - @Test public void add_error_1() { - HttpTest.execWithHttpException(HttpSC.BAD_REQUEST_400, - ()-> addTestDataset(fileBase+"config-ds-bad-name-1.ttl")); - } - - @Test public void add_error_2() { - HttpTest.execWithHttpException(HttpSC.BAD_REQUEST_400, - ()-> addTestDataset(fileBase+"config-ds-bad-name-2.ttl")); - } - - @Test public void add_error_3() { - HttpTest.execWithHttpException(HttpSC.BAD_REQUEST_400, - ()-> addTestDataset(fileBase+"config-ds-bad-name-3.ttl")); - } - - @Test public void add_error_4() { - HttpTest.execWithHttpException(HttpSC.BAD_REQUEST_400, - ()-> addTestDataset(fileBase+"config-ds-bad-name-4.ttl")); - } - - @Test public void delete_dataset_1() { - String name = "NoSuchDataset"; - HttpTest.expect404( ()-> httpDelete(ServerCtl.urlRoot()+"$/"+opDatasets+"/"+name) ); - } - -// // ---- Active/Offline. -// -// @Test public void state_1() { -// // Add one -// addTestDataset(); -// try { -// checkExists(dsTest); -// -// httpPost(ServerCtl.urlRoot()+"$/"+opDatasets+"/"+dsTest+"?state=offline"); -// -// checkExistsNotActive(dsTest); -// -// httpPost(ServerCtl.urlRoot()+"$/"+opDatasets+"/"+dsTest+"?state=active"); -// -// checkExists(dsTest); -// } finally { -// deleteDataset(dsTest); -// } -// } -// -// @Test public void state_2() { -// addTestDataset(); -// httpPost(ServerCtl.urlRoot()+"$/"+opDatasets+"/"+dsTest+"?state=offline"); -// deleteDataset(dsTest); -// checkNotThere(dsTest); -// } -// -// @Test public void state_3() { -// addTestDataset(); -// HttpTest.expect404(()->httpPost(ServerCtl.urlRoot()+"$/"+opDatasets+"/DoesNotExist?state=offline")); -// deleteDataset(dsTest); -// } - // ---- Backup @Test public void create_backup_1() { @@ -330,91 +178,6 @@ public void create_backup_2() { } } - // ---- Compact - - @Test public void compact_01() { - assumeNotWindows(); - - String testDB = dsTestTdb2b; - try { - checkNotThere(testDB); - addTestDatasetTDB2(testDB); - checkExists(testDB); - - String id = null; - try { - JsonValue v = httpPostRtnJSON(ServerCtl.urlRoot() + "$/" + opCompact + "/" + testDB); - id = v.getAsObject().getString(JsonConstCtl.taskId); - } finally { - waitForTasksToFinish(1000, 500, 20_000); - } - Assert.assertNotNull(id); - checkInTasks(id); - - JsonValue task = getTask(id); - // ---- - // The result assertion is throwing NPE occasionally on some heavily loaded CI servers. - // This may be because of server or test code encountering a very long wait. - // These next statements check the assumed structure of the return. - Assert.assertNotNull("Task value", task); - JsonObject obj = task.getAsObject(); - Assert.assertNotNull("Task.getAsObject()", obj); - // Provoke code to get a stacktrace. - obj.getBoolean(JsonConstCtl.success); - // ---- - // The assertion we really wanted to check. - // Check task success - Assert.assertTrue("Expected task to be marked as successful", task.getAsObject().getBoolean(JsonConstCtl.success)); - } finally { - deleteDataset(testDB); - } - } - - @Test public void compact_02() { - HttpTest.expect400(()->{ - JsonValue v = httpPostRtnJSON(ServerCtl.urlRoot() + "$/" + opCompact + "/noSuchDataset"); - }); - } - - private void assumeNotWindows() { - if (SystemUtils.IS_OS_WINDOWS) - throw new AssumptionViolatedException("Test may be unstable on Windows due to inability to delete memory-mapped files"); - } - - // ---- Server - - // ---- Stats - - @Test public void stats_1() { - JsonValue v = execGetJSON(ServerCtl.urlRoot()+"$/"+opStats); - checkJsonStatsAll(v); - } - - @Test public void stats_2() { - addTestDataset(); - JsonValue v = execGetJSON(ServerCtl.urlRoot()+"$/"+opStats+ServerCtl.datasetPath()); - checkJsonStatsAll(v); - deleteDataset(dsTest); - } - - @Test public void stats_3() { - addTestDataset(); - HttpTest.expect404(()-> execGetJSON(ServerCtl.urlRoot()+"$/"+opStats+"/DoesNotExist")); - deleteDataset(dsTest); - } - - @Test public void stats_4() { - JsonValue v = execPostJSON(ServerCtl.urlRoot()+"$/"+opStats); - checkJsonStatsAll(v); - } - - @Test public void stats_5() { - addTestDataset(); - JsonValue v = execPostJSON(ServerCtl.urlRoot()+"$/"+opStats+ServerCtl.datasetPath()); - checkJsonStatsAll(v); - deleteDataset(dsTest); - } - @Test public void sleep_1() { String x = execSleepTask(null, 1); } @@ -542,44 +305,6 @@ private static JsonValue getDatasetDescription(String dsName) { } } - // -- Add - - private static void addTestDataset() { - addTestDataset(fileBase+"config-ds-plain-1.ttl"); - } - - private static void addTestDatasetInf() { - addTestDataset(fileBase+"config-ds-inf.ttl"); - } - - private static void addTestDatasetTDB2(String DBname) { - Objects.nonNull(DBname); - if ( DBname.equals(dsTestTdb2a) ) { - addTestDataset(fileBase+"config-tdb2a.ttl"); - return; - } - if ( DBname.equals(dsTestTdb2b) ) { - addTestDataset(fileBase+"config-tdb2b.ttl"); - return; - } - throw new IllegalArgumentException("No configuration for "+DBname); - } - - private static void addTestDataset(String filename) { - try { - Path f = Path.of(filename); - BodyPublisher body = BodyPublishers.ofFile(f); - String ct = WebContent.contentTypeTurtle; - httpPost(ServerCtl.urlRoot()+"$/"+opDatasets, ct, body); - } catch (FileNotFoundException e) { - IO.exception(e); - } - } - - private static void deleteDataset(String name) { - httpDelete(ServerCtl.urlRoot()+"$/"+opDatasets+"/"+name); - } - private static String execSleepTask(String name, int millis) { String url = ServerCtl.urlRoot()+"$/sleep"; if ( name != null ) { @@ -703,22 +428,6 @@ private static void checkExists(String name) { askPing(name); } - private static void checkExistsNotActive(String name) { - adminPing(name); - try { askPing(name); - fail("askPing did not cause an Http Exception"); - } catch ( HttpException ex ) {} - JsonValue v = getDatasetDescription(name); - assertFalse(v.getAsObject().get("ds.state").getAsBoolean().value()); - } - - private static void checkNotThere(String name) { - String n = (name.startsWith("/")) ? name.substring(1) : name; - // Check gone exists. - HttpTest.expect404(()-> adminPing(n) ); - HttpTest.expect404(() -> askPing(n) ); - } - private static void checkJsonDatasetsAll(JsonValue v) { assertNotNull(v.getAsObject().get("datasets")); JsonArray a = v.getAsObject().get("datasets").getAsArray(); @@ -734,63 +443,5 @@ private static void checkJsonDatasetsOne(JsonValue v) { assertNotNull(obj.get("ds.state")); assertTrue(obj.get("ds.services").isArray()); } - - private static void checkJsonStatsAll(JsonValue v) { - assertNotNull(v.getAsObject().get("datasets")); - JsonObject a = v.getAsObject().get("datasets").getAsObject(); - for ( String dsname : a.keys() ) { - JsonValue obj = a.get(dsname).getAsObject(); - checkJsonStatsOne(obj); - } - } - - private static void checkJsonStatsOne(JsonValue v) { - checkJsonStatsCounters(v); - JsonObject obj1 = v.getAsObject().get("endpoints").getAsObject(); - for ( String srvName : obj1.keys() ) { - JsonObject obj2 = obj1.get(srvName).getAsObject(); - assertTrue(obj2.hasKey("description")); - assertTrue(obj2.hasKey("operation")); - checkJsonStatsCounters(obj2); - } - } - - private static void checkJsonStatsCounters(JsonValue v) { - JsonObject obj = v.getAsObject(); - assertTrue(obj.hasKey("Requests")); - assertTrue(obj.hasKey("RequestsGood")); - assertTrue(obj.hasKey("RequestsBad")); - } - - private static JsonValue execGetJSON(String url) { - try ( TypedInputStream in = httpGet(url) ) { - AssertExtra.assertEqualsIgnoreCase(WebContent.contentTypeJSON, in.getContentType()); - return JSON.parse(in); - } - } - - private static JsonValue execPostJSON(String url) { - try ( TypedInputStream in = httpPostStream(url, null, null, null) ) { - AssertExtra.assertEqualsIgnoreCase(WebContent.contentTypeJSON, in.getContentType()); - return JSON.parse(in); - } - } - - /* - GET /$/ping - POST /$/ping - POST /$/datasets/ - GET /$/datasets/ - DELETE /$/datasets/*{name}* - GET /$/datasets/*{name}* - POST /$/datasets/*{name}*?state=offline - POST /$/datasets/*{name}*?state=active - POST /$/backup/*{name}* - POST /$/compact/*{name}* - GET /$/server - POST /$/server/shutdown - GET /$/stats/ - GET /$/stats/*{name}* - */ }
jena-fuseki2/jena-fuseki-webapp/src/test/java/org/apache/jena/fuseki/TS_FusekiWebapp.java+2 −0 modified@@ -33,6 +33,8 @@ , TestWebappAuthUpdate_JDK.class , TestWebappFileUpload.class , TestWebappAdmin.class + , TestWebappAdminAddDeleteDatasetTemplate.class + , TestWebappAdminAddDeleteDatasetFile.class , TestWebappAdminAPI.class , TestWebappServerReadOnly.class , TestWebappMetrics.class
jena-fuseki2/jena-fuseki-webapp/testing/config-tdb2c.ttl+19 −0 added@@ -0,0 +1,19 @@ +## Licensed under the terms of http://www.apache.org/licenses/LICENSE-2.0 + +PREFIX : <#> +PREFIX fuseki: <http://jena.apache.org/fuseki#> +PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> + +PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#> +PREFIX ja: <http://jena.hpl.hp.com/2005/11/Assembler#> +PREFIX tdb2: <http://jena.apache.org/2016/tdb#> + +<#service1> rdf:type fuseki:Service ; + fuseki:name "test-tdb2b" ; + fuseki:endpoint [ fuseki:name "sparql" ; + fuseki:operation fuseki:query ] ; + fuseki:dataset <#dataset> . + +<#dataset> rdf:type tdb2:DatasetTDB2 ; + # Bad. + tdb2:location "../tdb2c" .
35350569b4c1GH-3288: Validate request parameters; refactor tests
16 files changed · +1636 −865
jena-fuseki2/jena-fuseki-core/src/test/java/org/apache/jena/fuseki/servlets/TestCrossOriginFilterMock.java+3 −3 modified@@ -23,8 +23,8 @@ import jakarta.servlet.ServletException; import jakarta.servlet.http.HttpServletRequest; import jakarta.servlet.http.HttpServletResponse; -import org.junit.Before; -import org.junit.Test; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; import java.io.IOException; import java.util.Collections; @@ -73,7 +73,7 @@ public Enumeration<String> getInitParameterNames() { HttpServletResponse httpServletResponse = mock(HttpServletResponse.class); FilterChain chain = mock(FilterChain.class); - @Before + @BeforeEach public void setUpTest() { when(httpServletRequest.getHeader("Origin")).thenReturn("http://localhost:12335"); when(httpServletRequest.getHeaders("Connection")).thenReturn(Collections.emptyEnumeration());
jena-fuseki2/jena-fuseki-main/src/main/java/org/apache/jena/fuseki/mgt/ActionDatasets.java+225 −79 modified@@ -20,8 +20,6 @@ import static java.lang.String.format; -import java.io.IOException; -import java.io.OutputStream; import java.io.StringReader; import java.nio.file.Files; import java.nio.file.Path; @@ -35,41 +33,46 @@ import org.apache.jena.atlas.json.JsonValue; import org.apache.jena.atlas.lib.FileOps; import org.apache.jena.atlas.lib.InternalErrorException; +import org.apache.jena.atlas.lib.NotImplemented; import org.apache.jena.atlas.logging.FmtLog; import org.apache.jena.atlas.web.ContentType; import org.apache.jena.datatypes.xsd.XSDDatatype; +import org.apache.jena.dboe.base.file.Location; +import org.apache.jena.fuseki.FusekiConfigException; import org.apache.jena.fuseki.build.DatasetDescriptionMap; import org.apache.jena.fuseki.build.FusekiConfig; import org.apache.jena.fuseki.ctl.ActionContainerItem; import org.apache.jena.fuseki.ctl.JsonDescription; import org.apache.jena.fuseki.metrics.MetricsProvider; -import org.apache.jena.fuseki.server.DataAccessPoint; -import org.apache.jena.fuseki.server.DataService; -import org.apache.jena.fuseki.server.FusekiVocab; -import org.apache.jena.fuseki.server.ServerConst; +import org.apache.jena.fuseki.server.*; import org.apache.jena.fuseki.servlets.ActionLib; import org.apache.jena.fuseki.servlets.HttpAction; import org.apache.jena.fuseki.servlets.ServletOps; -import org.apache.jena.fuseki.system.DataUploader; import org.apache.jena.fuseki.system.FusekiNetLib; +import org.apache.jena.graph.Graph; import org.apache.jena.graph.Node; +import org.apache.jena.query.Query; +import org.apache.jena.query.QueryFactory; import org.apache.jena.rdf.model.*; +import org.apache.jena.rdf.model.impl.Util; import org.apache.jena.riot.*; import org.apache.jena.riot.system.StreamRDF; import org.apache.jena.riot.system.StreamRDFLib; import org.apache.jena.shared.JenaException; import org.apache.jena.sparql.core.DatasetGraph; import org.apache.jena.sparql.core.Quad; import org.apache.jena.sparql.core.assembler.AssemblerUtils; +import org.apache.jena.sparql.exec.QueryExec; +import org.apache.jena.sparql.exec.RowSet; import org.apache.jena.sparql.util.FmtUtils; +import org.apache.jena.system.G; +import org.apache.jena.tdb1.TDB1; +import org.apache.jena.tdb2.TDB2; import org.apache.jena.vocabulary.RDF; import org.apache.jena.web.HttpSC; public class ActionDatasets extends ActionContainerItem { - - static private Property pServiceName = FusekiVocab.pServiceName; - //static private Property pStatus = FusekiVocab.pStatus; private static final String paramDatasetName = "dbName"; private static final String paramDatasetType = "dbType"; @@ -121,94 +124,160 @@ protected JsonValue execPostContainer(HttpAction action) { ServletOps.errorBadRequest("Bad request - Content-Type or both parameters dbName and dbType required"); boolean succeeded = false; - String systemFileCopy = null; + // Used in clear-up. String configFile = null; + String systemFileCopy = null; + FusekiServerCtl serverCtl = FusekiServerCtl.get(action.getServletContext()); DatasetDescriptionMap registry = new DatasetDescriptionMap(); - synchronized (FusekiAdmin.systemLock) { + synchronized (serverCtl.getServerlock()) { try { - // Where to build the templated service/database. - Model descriptionModel = ModelFactory.createDefaultModel(); - StreamRDF dest = StreamRDFLib.graph(descriptionModel.getGraph()); - - if ( hasParams || WebContent.isHtmlForm(ct) ) - assemblerFromForm(action, dest); - else if ( WebContent.isMultiPartForm(ct) ) - assemblerFromUpload(action, dest); - else - assemblerFromBody(action, dest); + // Get the request input. + Model modelFromRequest = ModelFactory.createDefaultModel(); + StreamRDF dest = StreamRDFLib.graph(modelFromRequest.getGraph()); - // ---- - // Keep a persistent copy immediately. This is not used for - // anything other than being "for the record". - systemFileCopy = FusekiServerCtl.dirSystemFileArea.resolve(uuid.toString()).toString(); - try ( OutputStream outCopy = IO.openOutputFile(systemFileCopy) ) { - RDFDataMgr.write(outCopy, descriptionModel, Lang.TURTLE); + boolean templatedRequest = false; + + try { + if ( hasParams || WebContent.isHtmlForm(ct) ) { + assemblerFromForm(action, dest); + templatedRequest = true; + // dbName, dbType + } else if ( WebContent.isMultiPartForm(ct) ) { + // Cannot be enabled. + ServletOps.errorBadRequest("Service configuration from a multipart upload not supported"); + //assemblerFromUpload(action, dest); + } else { + if ( ! FusekiAdmin.allowConfigFiles() ) + ServletOps.errorBadRequest("Service configuration from an upload file not supported"); + assemblerFromBody(action, dest); + } + } catch (RiotException ex) { + ActionLib.consumeBody(action); + action.log.warn(format("[%d] Failed to read configuration: %s", action.id, ex.getMessage())); + ServletOps.errorBadRequest("Failed to read configuration"); } // ---- // Add the dataset and graph wiring for assemblers Model model = ModelFactory.createDefaultModel(); - model.add(descriptionModel); + model.add(modelFromRequest); model = AssemblerUtils.prepareForAssembler(model); // ---- // Process configuration. - // Returns the "service fu:name NAME" statement Statement stmt = findService(model); + if ( stmt == null ) { + action.log.warn(format("[%d] No service name", action.id)); + ServletOps.errorBadRequest(format("No service name")); + } Resource subject = stmt.getSubject(); Literal object = stmt.getObject().asLiteral(); if ( object.getDatatype() != null && ! object.getDatatype().equals(XSDDatatype.XSDstring) ) action.log.warn(format("[%d] Service name '%s' is not a string", action.id, FmtUtils.stringForRDFNode(object))); - String datasetPath; - { // Check the name provided. + final String datasetPath; + { String datasetName = object.getLexicalForm(); // This duplicates the code FusekiBuilder.buildDataAccessPoint to give better error messages and HTTP status code." // ---- Check and canonicalize name. - if ( datasetName.isEmpty() ) - ServletOps.error(HttpSC.BAD_REQUEST_400, "Empty dataset name"); - if ( StringUtils.isBlank(datasetName) ) - ServletOps.error(HttpSC.BAD_REQUEST_400, format("Whitespace dataset name: '%s'", datasetName)); - if ( datasetName.contains(" ") ) - ServletOps.error(HttpSC.BAD_REQUEST_400, format("Bad dataset name (contains spaces) '%s'",datasetName)); - if ( datasetName.equals("/") ) - ServletOps.error(HttpSC.BAD_REQUEST_400, format("Bad dataset name '%s'",datasetName)); + // Various explicit check for better error messages. + + if ( datasetName.isEmpty() ) { + action.log.warn(format("[%d] Empty dataset name", action.id)); + ServletOps.errorBadRequest("Empty dataset name"); + } + if ( StringUtils.isBlank(datasetName) ) { + action.log.warn(format("[%d] Whitespace dataset name: '%s'", action.id, datasetName)); + ServletOps.errorBadRequest(format("Whitespace dataset name: '%s'", datasetName)); + } + if ( datasetName.contains(" ") ) { + action.log.warn(format("[%d] Bad dataset name (contains spaces) '%s'", action.id, datasetName)); + ServletOps.errorBadRequest(format("Bad dataset name (contains spaces) '%s'", datasetName)); + } + if ( datasetName.equals("/") ) { + action.log.warn(format("[%d] Bad dataset name '%s'", action.id, datasetName)); + ServletOps.errorBadRequest(format("Bad dataset name '%s'", datasetName)); + } + + // The service names must be a valid URI path + try { + ValidString validServiceName = Validators.serviceName(datasetName); + } catch (FusekiConfigException ex) { + action.log.warn(format("[%d] Invalid service name: '%s'", action.id, datasetName)); + ServletOps.error(HttpSC.BAD_REQUEST_400, format("Invalid service name: '%s'", datasetName)); + } + + // Canonical - starts with "/",does not end in "/" datasetPath = DataAccessPoint.canonical(datasetName); - // ---- Check whether it already exists - if ( action.getDataAccessPointRegistry().isRegistered(datasetPath) ) - ServletOps.error(HttpSC.CONFLICT_409, "Name already registered "+datasetPath); + + // For this operation, check additionally that the path does not go outside the expected file area. + // This imposes the path component-only rule and does not allow ".." + if ( ! isValidServiceName(datasetPath) ) { + action.log.warn(format("[%d] Database service name not acceptable: '%s'", action.id, datasetName)); + ServletOps.error(HttpSC.BAD_REQUEST_400, format("Database service name not acceptable: '%s'", datasetName)); + } + } + + // ---- Check whether it already exists + if ( action.getDataAccessPointRegistry().isRegistered(datasetPath) ) { + action.log.warn(format("[%d] Name already registered '%s'", action.id, datasetPath)); + ServletOps.error(HttpSC.CONFLICT_409, format("Name already registered '%s'", datasetPath)); } + // -- Validate any TDB locations. + // If this is a templated request, there is no need to do this + // because the location is "datasetPath" which has been checked. + if ( ! templatedRequest ) { + List<String> tdbLocations = tdbLocations(action, model.getGraph()); + for(String tdbLocation : tdbLocations ) { + if ( ! isValidTDBLocation(tdbLocation) ) { + action.log.warn(format("[%d] TDB database location not acceptable: '%s'", action.id, tdbLocation)); + ServletOps.error(HttpSC.BAD_REQUEST_400, format("TDB database location not acceptable: '%s'", tdbLocation)); + } + } + } + + // ---- + // Keep a persistent copy with a globally unique name. + // This is not used for anything other than being "for the record". + systemFileCopy = FusekiServerCtl.dirSystemFileArea.resolve(uuid.toString()).toString(); + RDFWriter.source(model).lang(Lang.TURTLE).output(systemFileCopy); + + // ---- action.log.info(format("[%d] Create database : name = %s", action.id, datasetPath)); - configFile = FusekiServerCtl.generateConfigurationFilename(datasetPath); List<String> existing = FusekiServerCtl.existingConfigurationFile(datasetPath); if ( ! existing.isEmpty() ) ServletOps.error(HttpSC.CONFLICT_409, "Configuration file for '"+datasetPath+"' already exists"); - // Write to configuration directory. - try ( OutputStream outCopy = IO.openOutputFile(configFile) ) { - RDFDataMgr.write(outCopy, descriptionModel, Lang.TURTLE); - } + configFile = FusekiServerCtl.generateConfigurationFilename(datasetPath); - // Need to be in Resource space at this point. + // ---- Build the service DataAccessPoint dataAccessPoint = FusekiConfig.buildDataAccessPoint(subject.getModel().getGraph(), subject.asNode(), registry); if ( dataAccessPoint == null ) { FmtLog.error(action.log, "Failed to build DataAccessPoint: datasetPath = %s; DataAccessPoint name = %s", datasetPath, dataAccessPoint); ServletOps.errorBadRequest("Failed to build DataAccessPoint"); return null; } dataAccessPoint.getDataService().setEndpointProcessors(action.getOperationRegistry()); - dataAccessPoint.getDataService().goActive(); + + // Write to configuration directory. + RDFWriter.source(model).lang(Lang.TURTLE).output(configFile); + if ( ! datasetPath.equals(dataAccessPoint.getName()) ) FmtLog.warn(action.log, "Inconsistent names: datasetPath = %s; DataAccessPoint name = %s", datasetPath, dataAccessPoint); + + dataAccessPoint.getDataService().goActive(); succeeded = true; + + // At this point, a server restarting will find the new service. + // This next line makes it dispatchable in this running server. action.getDataAccessPointRegistry().register(dataAccessPoint); // Add to metrics @@ -218,8 +287,8 @@ else if ( WebContent.isMultiPartForm(ct) ) action.setResponseContentType(WebContent.contentTypeTextPlain); ServletOps.success(action); - } catch (IOException ex) { IO.exception(ex); } - finally { + } finally { + // Clear-up on failure. if ( ! succeeded ) { if ( systemFileCopy != null ) FileOps.deleteSilent(systemFileCopy); if ( configFile != null ) FileOps.deleteSilent(configFile); @@ -229,6 +298,41 @@ else if ( WebContent.isMultiPartForm(ct) ) } } + /** + * Check whether a service name is acceptable. + * A service name is used as a filesystem path component, + * except it may have a leading "/"., to store the database and the configuration. + * <p> + * The canonical name for a service (see {@link DataAccessPoint#canonical}) + * starts with a "/" and this will be added if necessary. + */ + private boolean isValidServiceName(String datasetPath) { + // Leading "/" is OK , nowhere else is. + int idx = datasetPath.indexOf('/', 1); + if ( idx > 0 ) + return false; + // No slash, except maybe at the start so a meaningful use of .. can only be at the start. + if ( datasetPath.startsWith("/..")) + return false; + // Character restrictions done by Validators.serviceName + return true; + } + + // This works for TDB1 as well. + private boolean isValidTDBLocation(String tdbLocation) { + Location location = Location.create(tdbLocation); + if ( location.isMem() ) + return true; + // No ".." + if (tdbLocation.startsWith("..") || tdbLocation.contains("/..") ) { + // That test was too strict. + List<String> components = FileOps.pathComponents(tdbLocation); + if ( components.contains("..") ) + return false; + } + return true; + } + /** Find the service resource. There must be only one in the configuration. */ private Statement findService(Model model) { // Try to find by unique pServiceName (max backwards compatibility) @@ -261,8 +365,11 @@ private Statement findService(Model model) { stmt = stmt3; } + if ( stmt == null ) + return null; + if ( ! stmt.getObject().isLiteral() ) - ServletOps.errorBadRequest("Found "+FmtUtils.stringForRDFNode(stmt.getObject())+" : Service names are strings, then used to build the external URI"); + ServletOps.errorBadRequest("Found "+FmtUtils.stringForRDFNode(stmt.getObject())+" : Service names are strings, which are then used to build the external URI"); return stmt; } @@ -305,29 +412,27 @@ protected void execDeleteItem(HttpAction action) { ServletOps.errorNotFound("No such dataset registered: "+name); boolean succeeded = false; + FusekiServerCtl serverCtl = FusekiServerCtl.get(action.getServletContext()); - synchronized(FusekiAdmin.systemLock ) { + synchronized(serverCtl.getServerlock()) { try { - // Here, go offline. - // Need to reference count operations when they drop to zero - // or a timer goes off, we delete the dataset. - // Redo check inside transaction. DataAccessPoint ref = action.getDataAccessPointRegistry().get(name); if ( ref == null ) ServletOps.errorNotFound("No such dataset registered: "+name); // Get a reference before removing. DataService dataService = ref.getDataService(); - // ---- Make it invisible in this running server. + + // Remove from the registry - operation dispatch will not find it any more. action.getDataAccessPointRegistry().remove(name); // Find the configuration. String filename = name.startsWith("/") ? name.substring(1) : name; List<String> configurationFiles = FusekiServerCtl.existingConfigurationFile(filename); if ( configurationFiles.isEmpty() ) { - // ---- Unmanaged + // -- Unmanaged action.log.warn(format("[%d] Can't delete database configuration - not a managed database", action.id, name)); // ServletOps.errorOccurred(format("Can't delete database - not a managed configuration", name)); succeeded = true; @@ -343,23 +448,22 @@ protected void execDeleteItem(HttpAction action) { return; } - // ---- Remove managed database. + // -- Remove managed database. String cfgPathname = configurationFiles.get(0); // Delete configuration file. // Once deleted, server restart will not have the database. FileOps.deleteSilent(cfgPathname); - // Delete the database for real only when it is in the server "run/databases" - // area. Don't delete databases that reside elsewhere. We do delete the - // configuration file, so the databases will not be associated with the server - // anymore. + // Delete the database for real only if it is in the server + // "run/databases" area. Don't delete databases that reside + // elsewhere. We have already deleted the configuration file, so the + // databases will not be associated with the server anymore. @SuppressWarnings("removal") boolean isTDB1 = org.apache.jena.tdb1.sys.TDBInternal.isTDB1(dataService.getDataset()); boolean isTDB2 = org.apache.jena.tdb2.sys.TDBInternal.isTDB2(dataService.getDataset()); - // This occasionally fails in tests due to outstanding transactions. try { dataService.shutdown(); } catch (JenaException ex) { @@ -368,8 +472,9 @@ protected void execDeleteItem(HttpAction action) { // JENA-1481: Really delete files. if ( ( isTDB1 || isTDB2 ) ) { // Delete databases created by the UI, or the admin operation, which are - // in predictable, unshared location on disk. + // in predictable, unshared locations on disk. // There may not be any database files, the in-memory case. + // (TDB supports an in-memory mode.) Path pDatabase = FusekiServerCtl.dirDatabases.resolve(filename); if ( Files.exists(pDatabase)) { try { @@ -406,18 +511,16 @@ private static void assemblerFromBody(HttpAction action, StreamRDF dest) { } private static void assemblerFromForm(HttpAction action, StreamRDF dest) { - String x = action.getRequestQueryString(); String dbType = action.getRequestParameter(paramDatasetType); String dbName = action.getRequestParameter(paramDatasetName); - if ( StringUtils.isBlank(dbType) || StringUtils.isBlank(dbName) ) - ServletOps.errorBadRequest("Received HTML form. Both parameters 'dbName' and 'dbType' required"); + // Test for null, empty or only whitespace. + if ( StringUtils.isBlank(dbType) || StringUtils.isBlank(dbName) ) { + action.log.warn(format("[%d] Both parameters 'dbName' and 'dbType' required and not be blank", action.id)); + ServletOps.errorBadRequest("Received HTML form. Both parameters 'dbName' and 'dbType' required"); + } Map<String, String> params = new HashMap<>(); - - if ( dbName.startsWith("/") ) - params.put(Template.NAME, dbName.substring(1)); - else - params.put(Template.NAME, dbName); + params.put(Template.NAME, dbName); FusekiServerCtl serverCtl = FusekiServerCtl.get(action.getServletContext()); if ( serverCtl != null ) @@ -427,8 +530,7 @@ private static void assemblerFromForm(HttpAction action, StreamRDF dest) { // No return. } - //action.log.info(format("[%d] Create database : name = %s, type = %s", action.id, dbName, dbType )); - + // -- Get the template String template = dbTypeToTemplate.get(dbType.toLowerCase(Locale.ROOT)); if ( template == null ) { List<String> keys = new ArrayList<>(dbTypeToTemplate.keySet()); @@ -441,7 +543,8 @@ private static void assemblerFromForm(HttpAction action, StreamRDF dest) { } private static void assemblerFromUpload(HttpAction action, StreamRDF dest) { - DataUploader.incomingData(action, dest); + throw new NotImplemented(); + //DataUploader.incomingData(action, dest); } // ---- Auxiliary functions @@ -476,6 +579,49 @@ private static void bodyAsGraph(HttpAction action, StreamRDF dest) { return; } dest.prefix("root", base+"#"); - ActionLib.parseOrError(action, dest, lang, base); + ActionLib.parse(action, dest, lang, base); + } + + // ---- POST + + private static final String NL = "\n"; + + @SuppressWarnings("removal") + private static final String queryStringLocations = + "PREFIX tdb1: <"+TDB1.namespace+">"+NL+ + "PREFIX tdb2: <"+TDB2.namespace+">"+NL+ + """ + SELECT * { + ?x ( tdb2:location | tdb1:location) ?location + } + """ ; + + private static final Query queryLocations = QueryFactory.create(queryStringLocations); + + private static List<String> tdbLocations(HttpAction action, Graph configGraph) { + try ( QueryExec exec = QueryExec.graph(configGraph).query(queryLocations).build() ) { + RowSet results = exec.select(); + List<String> locations = new ArrayList<>(); + results.forEach(b->{ + Node loc = b.get("location"); + String location; + if ( loc.isURI() ) + location = loc.getURI(); + else if ( Util.isSimpleString(loc) ) + location = G.asString(loc); + else { + //action.log.warn(format("[%d] Database location is not a string nor a URI", action.id)); + // No return + ServletOps.errorBadRequest("TDB database location is not a string"); + location = null; + } + locations.add(location); + }); + return locations; + } catch (Exception ex) { + // No return + ServletOps.errorBadRequest("TDB database location can not be deterined"); + return null; + } } }
jena-fuseki2/jena-fuseki-main/src/main/java/org/apache/jena/fuseki/mgt/FusekiAdmin.java+17 −1 modified@@ -19,5 +19,21 @@ package org.apache.jena.fuseki.mgt; public class FusekiAdmin { - public final static Object systemLock = new Object(); + /** + * Control whether to allow creating new dataservices by uploading a config file. + * See {@link ActionDatasets}. + * + */ + public static final String allowConfigFileProperty = "fuseki:allowAddByConfigFile"; + + /** + * Return whether to allow service configuration files to be uploaded as a file. + * See {@link ActionDatasets}. + */ + public static boolean allowConfigFiles() { + String value = System.getProperty(allowConfigFileProperty); + if ( value != null ) + return "true".equals(value); + return false; + } }
jena-fuseki2/jena-fuseki-main/src/main/java/org/apache/jena/fuseki/mgt/FusekiServerCtl.java+4 −3 modified@@ -427,16 +427,17 @@ public static String generateConfigurationFilename(String dsName) { } /** Return the filenames of all matching files in the configuration directory (absolute paths returned ). */ - public static List<String> existingConfigurationFile(String baseFilename) { + public static List<String> existingConfigurationFile(String serviceName) { + String filename = DataAccessPoint.isCanonical(serviceName) ? serviceName.substring(1) : serviceName; try { List<String> paths = new ArrayList<>(); - try (DirectoryStream<Path> stream = Files.newDirectoryStream(FusekiServerCtl.dirConfiguration, baseFilename+".*") ) { + // This ".* is a file glob pattern, not a regular expression - it looks for file extensions. + try (DirectoryStream<Path> stream = Files.newDirectoryStream(FusekiServerCtl.dirConfiguration, filename+".*") ) { stream.forEach((p)-> paths.add(FusekiServerCtl.dirConfiguration.resolve(p).toString() )); } return paths; } catch (IOException ex) { throw new InternalErrorException("Failed to read configuration directory "+FusekiServerCtl.dirConfiguration); } } - }
jena-fuseki2/jena-fuseki-main/src/test/java/org/apache/jena/fuseki/main/FusekiTestLib.java+4 −0 modified@@ -43,6 +43,10 @@ public static void expect404(Runnable runnable) { expectFail(runnable, HttpSC.Code.NOT_FOUND); } + public static void expect409(Runnable runnable) { + expectFail(runnable, HttpSC.Code.CONFLICT); + } + public static void expectFail(Runnable runnable, Code code) { if ( code == null || ( 200 <= code.getCode() && code.getCode() < 300 ) ) { runnable.run();
jena-fuseki2/jena-fuseki-main/src/test/java/org/apache/jena/fuseki/mod/admin/FusekiServerPerTestClass.java+138 −0 added@@ -0,0 +1,138 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.jena.fuseki.mod.admin; + +import static org.junit.jupiter.api.Assertions.fail; + +import java.util.concurrent.TimeUnit; +import java.util.function.Consumer; + +import org.apache.jena.atlas.lib.FileOps; +import org.apache.jena.atlas.logging.LogCtl; +import org.apache.jena.fuseki.Fuseki; +import org.apache.jena.fuseki.ctl.ActionSleep; +import org.apache.jena.fuseki.main.FusekiServer; +import org.apache.jena.fuseki.main.sys.FusekiModules; +import org.apache.jena.fuseki.mgt.FusekiServerCtl; +import org.apache.jena.fuseki.mod.FusekiServerRunner; +import org.apache.jena.fuseki.server.DataAccessPointRegistry; +import org.apache.jena.fuseki.system.FusekiLogging; +import org.apache.jena.sparql.core.DatasetGraph; +import org.apache.jena.sparql.core.DatasetGraphFactory; +import org.awaitility.Awaitility; +import org.junit.jupiter.api.AfterAll; +import org.junit.jupiter.api.BeforeAll; + +/** + * Framework for running tests on a Fuseki server, with a single server for all tests. + */ +public class FusekiServerPerTestClass { + + private static String serverURL = null; + private static FusekiServer server = null; + + @BeforeAll public static void logging() { + FusekiLogging.setLogging(); + } + + @BeforeAll public static void startServer() { + System.setProperty("FUSEKI_BASE", serverArea()); + FileOps.clearAll(serverArea()); + + server = createServerForTest(); + serverURL = server.serverURL(); + } + + protected static String serverArea() { + return "target/run"; + } + + protected static DataAccessPointRegistry serverRegistry() { + return server.getDataAccessPointRegistry(); + } + + @AfterAll public static void stopServer() { + if ( server != null ) + server.stop(); + serverURL = null; + FusekiServerCtl.clearUpSystemState(); + } + + @BeforeAll public static void setLogging() { + LogCtl.setLevel(Fuseki.backupLogName, "ERROR"); + LogCtl.setLevel(Fuseki.compactLogName,"ERROR"); + Awaitility.setDefaultPollDelay(20,TimeUnit.MILLISECONDS); + Awaitility.setDefaultPollInterval(50,TimeUnit.MILLISECONDS); + } + + @AfterAll public static void unsetLogging() { + LogCtl.setLevel(Fuseki.backupLogName, "WARN"); + LogCtl.setLevel(Fuseki.compactLogName,"WARN"); + } + + // For the one-per-class setup, include the usual modules for jena-fuseki-server. + private static FusekiModules modulesSetup() { + return FusekiServerRunner.serverModules(); + } + + private static FusekiServer createServerForTest() { + FusekiModules modules = modulesSetup(); + DatasetGraph dsg = DatasetGraphFactory.createTxnMem(); + FusekiServer testServer = FusekiServer.create() + .fusekiModules(modules) + .port(0) + // Add a database. + .add(datasetName(), dsg) + // Action used for testing. + .addServlet("/$/sleep/*", new ActionSleep()) + .build() + .start(); + return testServer; + } + + protected static String urlRoot() { + return serverURL; + } + + protected static String adminURL() { + return serverURL + "$/"; + } + + protected static String datasetName() { + return "dataset"; + } + + protected FusekiServerPerTestClass() {} + + // One server per test. + + protected void withServer(Consumer<FusekiServer> action) { + action.accept(server); + } + + /** Expect two strings to be non-null and be {@link String#equalsIgnoreCase} */ + protected static void assertEqualsContectType(String expected, String actual) { + if ( expected == null && actual == null ) + return; + if ( expected == null || actual == null ) + fail("Expected: "+expected+" Got: "+actual); + if ( ! expected.equalsIgnoreCase(actual) ) + fail("Expected: "+expected+" Got: "+actual); + } +}
jena-fuseki2/jena-fuseki-main/src/test/java/org/apache/jena/fuseki/mod/admin/FusekiServerPerTest.java+109 −0 added@@ -0,0 +1,109 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.jena.fuseki.mod.admin; + +import static org.junit.jupiter.api.Assertions.fail; + +import java.util.concurrent.TimeUnit; +import java.util.function.Consumer; + +import org.apache.jena.atlas.lib.FileOps; +import org.apache.jena.atlas.logging.LogCtl; +import org.apache.jena.fuseki.Fuseki; +import org.apache.jena.fuseki.main.FusekiServer; +import org.apache.jena.fuseki.main.sys.FusekiModules; +import org.apache.jena.fuseki.mgt.FusekiServerCtl; +import org.apache.jena.fuseki.system.FusekiLogging; +import org.apache.jena.sparql.core.DatasetGraph; +import org.apache.jena.sparql.core.DatasetGraphFactory; +import org.awaitility.Awaitility; +import org.junit.jupiter.api.AfterAll; +import org.junit.jupiter.api.BeforeAll; +import org.junit.jupiter.api.BeforeEach; + +/** + * Framework for running tests on a Fuseki server, with a fresh server for each test. + */ +public class FusekiServerPerTest { + + @BeforeAll public static void logging() { + FusekiLogging.setLogging(); + } + + protected FusekiServerPerTest() {} + + // One server per test. + + protected void withServer(Consumer<FusekiServer> action) { + withServer(null, action); + } + + protected void withServer(String configFile, Consumer<FusekiServer> action) { + FusekiModules modules = modulesSetup(); + DatasetGraph dsg = DatasetGraphFactory.createTxnMem(); + FusekiServer.Builder builder = FusekiServer.create().port(0); + + if ( modules != null ) + builder.fusekiModules(modules); + + if ( configFile != null ) + builder.parseConfigFile(configFile); + + customizerServer(builder); + + FusekiServer testServer = builder.start(); + try { + action.accept(testServer); + } finally { + testServer.stop(); + FusekiServerCtl.clearUpSystemState(); + } + } + + protected void customizerServer(FusekiServer.Builder builder) {} + + protected FusekiModules modulesSetup() { return null; } + + @BeforeEach public void cleanStart() { + System.setProperty("FUSEKI_BASE", "target/run"); + FileOps.clearAll("target/run"); + } + + @BeforeAll public static void setLogging() { + LogCtl.setLevel(Fuseki.backupLogName, "ERROR"); + LogCtl.setLevel(Fuseki.compactLogName,"ERROR"); + Awaitility.setDefaultPollDelay(20,TimeUnit.MILLISECONDS); + Awaitility.setDefaultPollInterval(50,TimeUnit.MILLISECONDS); + } + + @AfterAll public static void unsetLogging() { + LogCtl.setLevel(Fuseki.backupLogName, "WARN"); + LogCtl.setLevel(Fuseki.compactLogName,"WARN"); + } + + /** Expect two strings to be non-null and be {@link String#equalsIgnoreCase} */ + protected static void assertEqualsContectType(String expected, String actual) { + if ( expected == null && actual == null ) + return; + if ( expected == null || actual == null ) + fail("Expected: "+expected+" Got: "+actual); + if ( ! expected.equalsIgnoreCase(actual) ) + fail("Expected: "+expected+" Got: "+actual); + } +}
jena-fuseki2/jena-fuseki-main/src/test/java/org/apache/jena/fuseki/mod/admin/TestAdminAddDatasetsConfigFile.java+318 −0 added@@ -0,0 +1,318 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.jena.fuseki.mod.admin; + +import static org.apache.jena.fuseki.mgt.ServerMgtConst.opDatasets; +import static org.apache.jena.http.HttpOp.httpDelete; +import static org.apache.jena.http.HttpOp.httpGet; +import static org.apache.jena.http.HttpOp.httpPost; +import static org.junit.jupiter.api.Assumptions.assumeFalse; + +import java.io.FileNotFoundException; +import java.net.http.HttpRequest.BodyPublisher; +import java.net.http.HttpRequest.BodyPublishers; +import java.nio.file.Path; +import java.util.Objects; +import java.util.function.Consumer; + +import org.apache.commons.lang3.SystemUtils; +import org.apache.jena.atlas.io.IO; +import org.apache.jena.atlas.logging.LogCtl; +import org.apache.jena.atlas.web.TypedInputStream; +import org.apache.jena.fuseki.Fuseki; +import org.apache.jena.fuseki.main.FusekiServer; +import org.apache.jena.fuseki.main.FusekiTestLib; +import org.apache.jena.fuseki.main.sys.FusekiModules; +import org.apache.jena.fuseki.mgt.FusekiAdmin; +import org.apache.jena.fuseki.test.HttpTest; +import org.apache.jena.riot.WebContent; +import org.apache.jena.web.HttpSC; +import org.junit.jupiter.api.AfterAll; +import org.junit.jupiter.api.BeforeAll; +import org.junit.jupiter.api.Test; + +/** + * Tests of the admin functionality adding and deleting datasets dynamically. + * See also {@link TestAdminAddDatasetTemplate}. + */ +public class TestAdminAddDatasetsConfigFile extends FusekiServerPerTest { + // Name of the dataset in the assembler file. + static String dsTest = "test-ds1"; + static String dsTestInf = "test-ds4"; + + static final String dsTestTdb2a = "test-tdb2a"; + static final String dsTestTdb2b = "test-tdb2b"; + static final String dsTestTdb2c = "test-tdb2c"; + static String fileBase = "testing/Config/"; + + // Exactly the module under test + @Override + protected FusekiModules modulesSetup() { + return FusekiModules.create(FMod_Admin.create()); + } + + @BeforeAll public static void loggingAdmin() { + LogCtl.setLevel(Fuseki.adminLogName, "ERROR"); + } + + @AfterAll public static void resetLoggingAdmin() { + LogCtl.setLevel(Fuseki.adminLogName, "ERROR"); + } + + protected void withServerFileEnabled(Consumer<FusekiServer> action) { + System.setProperty(FusekiAdmin.allowConfigFileProperty, "true"); + try { + super.withServer(null, action); + } finally { + System.getProperties().remove(FusekiAdmin.allowConfigFileProperty); + } + } + + @Test public void add_block_dataset_1() { + // Blocked by default. + withServer(server -> { + FusekiTestLib.expect400(()->addTestDatasetByFile(server, "config-ds-plain-1.ttl")); + }); + } + + @Test public void add_unblocked_dataset_1() { + // Blocked by default. + withServerFileEnabled(server -> { + addTestDatasetByFile(server, "config-ds-plain-1.ttl"); + }); + } + + // Try to add twice + @Test public void add_add_dataset_1() { + withServerFileEnabled(server -> { + checkNotThere(server, dsTest); + + addTestDatasetByFile(server, "config-ds-plain-1.ttl"); + checkExists(server, dsTest); + + // Second try should fail. + FusekiTestLib.expect409(()->addTestDatasetByFile(server, "config-ds-plain-1.ttl")); + + // Check still exists. + checkExists(server, dsTest); + // Delete-able. + deleteDataset(server, dsTest); + checkNotThere(server, dsTest); + }); + } + + @Test public void add_delete_dataset_1() { + withServerFileEnabled(server -> { + + checkNotThere(server, dsTest); + checkNotThere(server, dsTestInf); + addTestDatasetByFile(server, "config-ds-inf.ttl"); + checkNotThere(server, dsTest); + checkExists(server, dsTestInf); + + deleteDataset(server, dsTestInf); + + checkNotThere(server, dsTestInf); + addTestDatasetByFile(server, "config-ds-inf.ttl"); + checkExists(server, dsTestInf); + deleteDataset(server, dsTestInf); + }); + } + + @Test public void add_delete_dataset_2() { + withServerFileEnabled(server -> { + // New style operations : cause two fuseki:names + addTestDatasetByFile(server, "config-ds-plain-2.ttl"); + checkExists(server, "test-ds2"); + }); + } + + @Test public void add_delete_dataset_TDB_1() { + withServerFileEnabled(server -> { + + String testDB = dsTestTdb2a; + assumeNotWindows(); + + checkNotThere(server, testDB); + + addTestDatasetTDB2(server, testDB); + + // Check exists. + checkExists(server, testDB); + + // Remove it. + deleteDataset(server, testDB); + checkNotThere(server, testDB); + }); + } + + @Test public void add_delete_dataset_TDB_2() { + withServerFileEnabled(server -> { + // This has location "--mem--" + String testDB = dsTestTdb2b; + checkNotThere(server, testDB); + addTestDatasetTDB2(server, testDB); + // Check exists. + checkExists(server, testDB); + // Remove it. + deleteDataset(server, testDB); + checkNotThere(server, testDB); + }); + } + + @Test public void add_dataset_error_1() { + withServerFileEnabled(server -> { + HttpTest.execWithHttpException(HttpSC.BAD_REQUEST_400, + ()-> addTestDatasetByFile(server, "config-ds-bad-name-1.ttl")); + }); + } + + @Test public void add_dataset_error_2() { + withServerFileEnabled(server -> { + HttpTest.execWithHttpException(HttpSC.BAD_REQUEST_400, + ()-> addTestDatasetByFile(server, "config-ds-bad-name-2.ttl")); + }); + } + + @Test public void add_dataset_error_3() { + withServerFileEnabled(server -> { + HttpTest.execWithHttpException(HttpSC.BAD_REQUEST_400, + ()-> addTestDatasetByFile(server, "config-ds-bad-name-3.ttl")); + }); + } + + @Test public void add_dataset_error_4() { + withServerFileEnabled(server -> { + HttpTest.execWithHttpException(HttpSC.BAD_REQUEST_400, + ()-> addTestDatasetByFile(server, "config-ds-bad-name-4.ttl")); + }); + } + + @Test public void add_dataset_error_5() { + withServerFileEnabled(server -> { + HttpTest.execWithHttpException(HttpSC.BAD_REQUEST_400, + ()-> addTestDatasetBodyPublisher(server, BodyPublishers.noBody())); + }); + } + + @Test public void add_dataset_error_6() { + withServerFileEnabled(server -> { + HttpTest.execWithHttpException(HttpSC.BAD_REQUEST_400, + ()-> addTestDatasetBodyPublisher(server, BodyPublishers.ofString(""))); + }); + } + + @Test public void add_dataset_error_7() { + withServerFileEnabled(server -> { + String level = LogCtl.getLevel(Fuseki.adminLog); + LogCtl.setLevel(Fuseki.adminLog, "FATAL"); + HttpTest.execWithHttpException(HttpSC.BAD_REQUEST_400, + ()-> addTestDatasetBodyPublisher(server, BodyPublishers.ofString("JUNK"))); + LogCtl.setLevel(Fuseki.adminLog, level); + }); + } + + @Test public void add_dataset_error_8() { + withServerFileEnabled(server -> { + String level = LogCtl.getLevel(Fuseki.adminLog); + LogCtl.setLevel(Fuseki.adminLog, "FATAL"); + HttpTest.execWithHttpException(HttpSC.BAD_REQUEST_400, + ()-> addTestDatasetTDB2(server, dsTestTdb2c)); + LogCtl.setLevel(Fuseki.adminLog, level); + }); + } + + @Test public void delete_dataset_1() { + withServerFileEnabled(server -> { + String name = "NoSuchDataset"; + HttpTest.expect404( ()-> httpDelete(server.serverURL()+"$/"+opDatasets+"/"+name) ); + }); + } + + // ---- Backup + + private void assumeNotWindows() { + assumeFalse(SystemUtils.IS_OS_WINDOWS, "Test may be unstable on Windows due to inability to delete memory-mapped files"); + } + + private void deleteDataset(FusekiServer server, String name) { + httpDelete(server.serverURL()+"$/"+opDatasets+"/"+name); + } + + private void addTestDatasetWithName(FusekiServer server, String dsName) { + addTestDatasetWithName(server, dsName, "mem"); + } + + private void addTestDatasetWithName(FusekiServer server, String dsName, String dbType) { + String URL = server.serverURL()+"$/"+opDatasets+"?dbName="+dsName+"&dbType="+dbType; + String ct = WebContent.contentTypeTurtle; + httpPost(URL); + } + + private void addTestDatasetTDB2(FusekiServer server, String DBname) { + Objects.nonNull(DBname); + switch(DBname) { + case dsTestTdb2a-> addTestDatasetByFile(server, "config-tdb2a.ttl"); + case dsTestTdb2b-> addTestDatasetByFile(server, "config-tdb2b.ttl"); + case dsTestTdb2c-> addTestDatasetByFile(server, "config-tdb2c.ttl"); + default->throw new IllegalArgumentException("No configuration for "+DBname); + } + } + + private void addTestDatasetByFile(FusekiServer server, String filename) { + try { + Path f = Path.of(fileBase+filename); + BodyPublisher body = BodyPublishers.ofFile(f); + addTestDatasetBodyPublisher(server, body); + } catch (FileNotFoundException e) { + IO.exception(e); + } + } + + private void addTestDatasetBodyPublisher(FusekiServer server, BodyPublisher body) { + String ct = WebContent.contentTypeTurtle; + httpPost(server.serverURL()+"$/"+opDatasets, ct, body); + } + + private void askPing(FusekiServer server, String name) { + if ( name.startsWith("/") ) + name = name.substring(1); + try ( TypedInputStream in = httpGet(server.serverURL()+name+"/sparql?query=ASK%7B%7D") ) { + IO.skipToEnd(in); + } + } + + private void adminPing(FusekiServer server, String name) { + try ( TypedInputStream in = httpGet(server.serverURL()+"$/"+opDatasets+"/"+name) ) { + IO.skipToEnd(in); + } + } + + private void checkExists(FusekiServer server, String name) { + adminPing(server, name); + askPing(server, name); + } + + private void checkNotThere(FusekiServer server, String name) { + String n = (name.startsWith("/")) ? name.substring(1) : name; + // Check gone exists. + HttpTest.expect404(()-> adminPing(server, n)); + HttpTest.expect404(() -> askPing(server, n)); + } +}
jena-fuseki2/jena-fuseki-main/src/test/java/org/apache/jena/fuseki/mod/admin/TestAdminAddDatasetTemplate.java+241 −0 added@@ -0,0 +1,241 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.jena.fuseki.mod.admin; + +import static org.apache.jena.fuseki.mgt.ServerMgtConst.opDatasets; +import static org.apache.jena.http.HttpOp.httpPost; +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.junit.jupiter.api.Assertions.assertTrue; + +import java.io.IOException; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.Paths; + +import org.apache.jena.atlas.io.IOX; +import org.apache.jena.atlas.logging.LogCtl; +import org.apache.jena.atlas.web.HttpException; +import org.apache.jena.atlas.web.TypedInputStream; +import org.apache.jena.base.Sys; +import org.apache.jena.fuseki.Fuseki; +import org.apache.jena.fuseki.main.FusekiTestLib; +import org.apache.jena.fuseki.mgt.FusekiServerCtl; +import org.apache.jena.http.HttpOp; +import org.apache.jena.query.QueryExecution; +import org.apache.jena.rdfconnection.RDFConnection; +import org.apache.jena.riot.WebContent; +import org.apache.jena.sparql.exec.http.Params; +import org.apache.jena.web.HttpSC; +import org.junit.jupiter.api.AfterAll; +import org.junit.jupiter.api.BeforeAll; +import org.junit.jupiter.api.Test; + +/** + * Tests of the admin functionality on an empty server and using the template mechanism. + * See also {@link TestAdmin}. + */ +public class TestAdminAddDatasetTemplate extends FusekiServerPerTestClass { + + @BeforeAll public static void loggingAdmin() { + LogCtl.setLevel(Fuseki.adminLogName, "ERROR"); + } + + @AfterAll public static void resetLoggingAdmin() { + LogCtl.setLevel(Fuseki.adminLogName, "ERROR"); + } + + @Test public void add_dataset_01() { + testAddDataset("db_1"); + } + + @Test public void add_dataset_02() { + testAddDataset( "/db_2"); + } + + @Test public void add_dataset_bad_01() { + String dbName = "db_bad_01"; + testAddDataset(dbName); + // This should fail. + FusekiTestLib.expect409(()->testAddDataset(dbName)); + } + + @Test public void add_dataset_bad_02() { + badAddDataserverRequest("bad_2 illegal"); + } + + @Test public void add_dataset_bad_03() { + badAddDataserverRequest("bad_3/path"); + } + + @Test public void add_dataset_bad_04() { + badAddDataserverRequest(""); + } + + @Test public void add_dataset_bad_05() { + badAddDataserverRequest(" "); + } + + @Test public void add_dataset_bad_06() { + badAddDataserverRequest("bad_6_AB CD"); + } + + @Test public void add_dataset_bad_07() { + badAddDataserverRequest(".."); + } + + @Test public void add_dataset_bad_08() { + badAddDataserverRequest("/.."); + } + + @Test public void add_dataset_bad_09() { + badAddDataserverRequest("/../elsewhere"); + } + + @Test public void add_dataset_bad_10() { + badAddDataserverRequest("//bad_10"); + } + + //@Test + public void noOverwriteExistingConfigFile() { + withServer(server->{ + try { + var workingDir = Paths.get(serverArea()).toAbsolutePath(); + var path = workingDir.resolve("configuration/test-ds0-empty.ttl"); + var dbConfig = path.toFile(); + dbConfig.createNewFile(); + try { + // refresh the file system so that the file exists + dbConfig = path.toFile(); + assertTrue (dbConfig.exists()); + assertEquals(0, dbConfig.length()); + + // Try to override the file with a new configuration. + String ct = WebContent.contentTypeHTMLForm; + String body = "dbName=test-ds0-empty&dbType=mem"; + HttpException ex = assertThrows(org.apache.jena.atlas.web.HttpException.class, + ()-> httpPost(server.serverURL() + "$/" + opDatasets, ct, body)); + assertEquals(HttpSC.CONFLICT_409, ex.getStatusCode()); + // refresh the file system + dbConfig = path.toFile(); + assertTrue(dbConfig.exists()); + assertEquals(0, dbConfig.length(), "File should be still empty"); + } + finally { + // Clean up the file. + if (Files.exists(path)) { + Files.delete(path); + } + } + } catch (IOException ex) { throw IOX.exception(ex); } + }); + } + + // add-delete + + @Test public void add_delete_mem_1() { + testAddDeleteAdd("db_add_delete_1", "mem", false, false); + } + + @Test public void add_delete_tdb_1() { + if ( Sys.isWindows ) + return; + testAddDeleteAdd("db_add_delete_tdb_1", "tdb2", false, true); + } + + @Test public void add_delete_tdb_2() { + if ( Sys.isWindows ) + return; + String dbName = "db_add_delete_tdb_2"; + testAddDeleteAdd(dbName, "tdb2", false, true); + } + + // Attempt to add a in-memory dataset. Used to test the name checking. + private void testAddDataset(String dbName) { + withServer(server->{ + String datasetURL = server.datasetURL(dbName); + Params params = Params.create().add("dbName", dbName).add("dbType", "mem"); + // Use the template + HttpOp.httpPostForm(adminURL()+"datasets", params); + assertTrue(exists(datasetURL)); + }); + } + + private void testAddDeleteAdd(String dbName, String dbType, boolean alreadyExists, boolean hasFiles) { + withServer(server->{ + String datasetURL = server.datasetURL(dbName); + Params params = Params.create().add("dbName", dbName).add("dbType", dbType); + + if ( alreadyExists ) + assertTrue(exists(datasetURL)); + else + assertFalse(exists(datasetURL)); + + // Use the template + HttpOp.httpPostForm(adminURL()+"datasets", params); + + RDFConnection conn = RDFConnection.connect(server.datasetURL(dbName)); + conn.update("INSERT DATA { <x:s> <x:p> 123 }"); + int x1 = count(conn); + assertEquals(1, x1); + + Path pathDB = FusekiServerCtl.dirDatabases.resolve(dbName); + + if ( hasFiles ) + assertTrue(Files.exists(pathDB)); + + HttpOp.httpDelete(adminURL()+"datasets/"+dbName); + + assertFalse(exists(datasetURL)); + + //if ( hasFiles ) + assertFalse(Files.exists(pathDB)); + + // Recreate : no contents. + HttpOp.httpPostForm(adminURL()+"datasets", params); + assertTrue(exists(datasetURL), ()->"false: exists("+datasetURL+")"); + int x2 = count(conn); + assertEquals(0, x2); + if ( hasFiles ) + assertTrue(Files.exists(pathDB)); + }); + } + + private void badAddDataserverRequest(String dbName) { + FusekiTestLib.expect400(()->testAddDataset(dbName)); + } + + private static boolean exists(String url) { + try ( TypedInputStream in = HttpOp.httpGet(url) ) { + return true; + } catch (HttpException ex) { + if ( ex.getStatusCode() == HttpSC.NOT_FOUND_404 ) + return false; + throw ex; + } + } + + static int count(RDFConnection conn) { + try ( QueryExecution qExec = conn.query("SELECT (count(*) AS ?C) { ?s ?p ?o }")) { + return qExec.execSelect().next().getLiteral("C").getInt(); + } + } +} +
jena-fuseki2/jena-fuseki-main/src/test/java/org/apache/jena/fuseki/mod/admin/TestAdminDatabaseOps.java+484 −0 added@@ -0,0 +1,484 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.jena.fuseki.mod.admin; + +import static org.apache.jena.fuseki.mgt.ServerMgtConst.opBackup; +import static org.apache.jena.fuseki.mgt.ServerMgtConst.opCompact; +import static org.apache.jena.fuseki.mgt.ServerMgtConst.opDatasets; +import static org.apache.jena.fuseki.mgt.ServerMgtConst.opListBackups; +import static org.apache.jena.fuseki.server.ServerConst.opStats; +import static org.apache.jena.http.HttpOp.*; +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertTrue; +import static org.junit.jupiter.api.Assertions.fail; +import static org.junit.jupiter.api.Assumptions.assumeFalse; + +import java.util.ArrayList; +import java.util.List; +import java.util.Objects; + +import org.apache.commons.lang3.SystemUtils; +import org.apache.jena.atlas.io.IO; +import org.apache.jena.atlas.json.JSON; +import org.apache.jena.atlas.json.JsonArray; +import org.apache.jena.atlas.json.JsonObject; +import org.apache.jena.atlas.json.JsonValue; +import org.apache.jena.atlas.lib.Lib; +import org.apache.jena.atlas.web.TypedInputStream; +import org.apache.jena.fuseki.ctl.JsonConstCtl; +import org.apache.jena.fuseki.main.FusekiServer; +import org.apache.jena.fuseki.main.sys.FusekiModules; +import org.apache.jena.fuseki.system.FusekiLogging; +import org.apache.jena.fuseki.test.HttpTest; +import org.apache.jena.riot.WebContent; +import org.apache.jena.sparql.core.DatasetGraphFactory; +import org.junit.jupiter.api.BeforeAll; +import org.junit.jupiter.api.Test; + +/** + * Tests of the admin functionality adding and deleting datasets dynamically. + * See also {@link TestAdminAddDatasetTemplate}. + */ +public class TestAdminDatabaseOps extends FusekiServerPerTest { + // Name of the dataset in the assembler file. + private static String dsTest = "test-ds1"; + + @BeforeAll public static void logging() { + FusekiLogging.setLogging(); + } + + @Override + protected FusekiModules modulesSetup() { + return FusekiModules.create(FMod_Admin.create()); + } + + @Override + protected void customizerServer(FusekiServer.Builder builder) { + builder.add(datasetName(), DatasetGraphFactory.createTxnMem()); + } + + private String datasetName() { + return "dsg"; + } + + // ---- Backup + + @Test public void create_backup_1() { + withServer(server -> { + String id = null; + try { + JsonValue v = httpPostRtnJSON(server.serverURL() + "$/" + opBackup + "/" + datasetName()); + id = v.getAsObject().getString("taskId"); + } finally { + waitForTasksToFinish(server, 1000, 10, 20000); + } + assertNotNull(id); + checkInTasks(server, id); + + // Check a backup was created + try ( TypedInputStream in = httpGet(server.serverURL()+"$/"+opListBackups) ) { + assertEqualsContectType(WebContent.contentTypeJSON, in.getContentType()); + JsonValue v = JSON.parseAny(in); + assertNotNull(v.getAsObject().get("backups")); + JsonArray a = v.getAsObject().get("backups").getAsArray(); + assertEquals(1, a.size()); + } + + JsonValue task = getTask(server, id); + assertNotNull(id); + // Expect task success + assertTrue(task.getAsObject().getBoolean(JsonConstCtl.success), "Expected task to be marked as successful"); + }); + } + + @Test public void create_backup_2() { + withServer(server -> { + HttpTest.expect400(()->{ + JsonValue v = httpPostRtnJSON(server.serverURL() + "$/" + opBackup + "/noSuchDataset"); + }); + }); + } + + @Test public void list_backups_1() { + withServer(server -> { + try ( TypedInputStream in = httpGet(server.serverURL()+"$/"+opListBackups) ) { + assertEqualsContectType(WebContent.contentTypeJSON, in.getContentType()); + JsonValue v = JSON.parseAny(in); + assertNotNull(v.getAsObject().get("backups")); + } + }); + } + + // ---- Compact + + @Test public void compact_01() { + withServer(server -> { + assumeNotWindows(); + + String testDB = "dsg-tdb2"; + try { + checkNotThere(server, testDB); + addTestDatasetTDB2(server, testDB); + checkExists(server, testDB); + + String id = null; + try { + JsonValue v = httpPostRtnJSON(server.serverURL() + "$/" + opCompact + "/" + testDB); + id = v.getAsObject().getString(JsonConstCtl.taskId); + } finally { + waitForTasksToFinish(server, 1000, 500, 20_000); + } + assertNotNull(id); + checkInTasks(server, id); + + JsonValue task = getTask(server, id); + // ---- + // The result assertion is throwing NPE occasionally on some heavily loaded CI servers. + // This may be because of server or test code encountering a very long wait. + // These next statements check the assumed structure of the return. + assertNotNull(task, "Task value"); + JsonObject obj = task.getAsObject(); + assertNotNull(obj, "Task.getAsObject()"); + // Provoke code to get a stacktrace. + obj.getBoolean(JsonConstCtl.success); + // ---- + // The assertion we really wanted to check. + // Check task success + assertTrue(task.getAsObject().getBoolean(JsonConstCtl.success), + "Expected task to be marked as successful"); + } finally { + deleteDataset(server, testDB); + } + }); + } + + @Test public void compact_02() { + withServer(server -> { + HttpTest.expect400(()->{ + JsonValue v = httpPostRtnJSON(server.serverURL() + "$/" + opCompact + "/noSuchDataset"); + }); + }); + } + + private void assumeNotWindows() { + assumeFalse(SystemUtils.IS_OS_WINDOWS, "Test may be unstable on Windows due to inability to delete memory-mapped files"); + } + + // ---- Server + + // ---- Stats + + @Test public void stats_1() { + withServer(server -> { + JsonValue v = execGetJSON(server.serverURL()+"$/"+opStats); + checkJsonStatsAll(v); + }); + } + + @Test public void stats_2() { + withServer(server -> { + addTestDatasetWithName(server, dsTest); + JsonValue v = execGetJSON(server.serverURL()+"$/"+opStats+"/"+dsTest); + checkJsonStatsAll(v); + deleteDataset(server, dsTest); + }); + } + + @Test public void stats_3() { + withServer(server -> { + addTestDatasetWithName(server, dsTest); + HttpTest.expect404(()-> execGetJSON(server.serverURL()+"$/"+opStats+"/DoesNotExist")); + deleteDataset(server, dsTest); + }); + } + + @Test public void stats_4() { + withServer(server -> { + JsonValue v = execPostJSON(server.serverURL()+"$/"+opStats); + checkJsonStatsAll(v); + }); + } + + @Test public void stats_5() { + withServer(server -> { + addTestDatasetWithName(server, dsTest); + JsonValue v = execPostJSON(server.serverURL()+"$/"+opStats+"/"+dsTest); + checkJsonStatsAll(v); + deleteDataset(server, dsTest); + }); + } + + // --- List all datasets + + @Test public void list_datasets_1() { + withServer(server->{ + try ( TypedInputStream in = httpGet(urlRoot(server)+"$/"+opDatasets); ) { + IO.skipToEnd(in); + } + }); + } + + @Test public void list_datasets_2() { + withServer(server->{ + try ( TypedInputStream in = httpGet(urlRoot(server)+"$/"+opDatasets) ) { + assertEqualsContentType(WebContent.contentTypeJSON, in.getContentType()); + JsonValue v = JSON.parseAny(in); + assertNotNull(v.getAsObject().get("datasets")); + checkJsonDatasetsAll(v); + } + }); + } + + // Specific dataset + @Test public void list_datasets_3() { + withServer( server->checkExists(server, datasetName()) ); + } + + // Specific dataset + @Test public void list_datasets_4() { + withServer( server->{ + HttpTest.expect404( () -> getDatasetDescription(server, "does-not-exist") ); + }); + } + + // Specific dataset + @Test public void list_datasets_5() { + withServer( server->{ + JsonValue v = getDatasetDescription(server, datasetName()); + checkJsonDatasetsOne(v.getAsObject()); + }); + } + + private String urlRoot(FusekiServer server) { + return server.serverURL(); + } + + private void deleteDataset(FusekiServer server, String name) { + httpDelete(server.serverURL()+"$/"+opDatasets+"/"+name); + } + + private JsonValue getTask(FusekiServer server, String taskId) { + String url = server.serverURL()+"$/tasks/"+taskId; + return httpGetJson(url); + } + + private JsonValue getDatasetDescription(FusekiServer server, String dsName) { + if ( dsName.startsWith("/") ) + dsName = dsName.substring(1); + try (TypedInputStream in = httpGet(server.serverURL() + "$/" + opDatasets + "/" + dsName)) { + assertEqualsContectType(WebContent.contentTypeJSON, in.getContentType()); + JsonValue v = JSON.parse(in); + return v; + } + } + + private void addTestDatasetWithName(FusekiServer server, String dsName) { + addTestDatasetWithName(server, dsName, "mem"); + } + + private void addTestDatasetWithName(FusekiServer server, String dsName, String dbType) { + String URL = server.serverURL()+"$/"+opDatasets+"?dbName="+dsName+"&dbType="+dbType; + String ct = WebContent.contentTypeTurtle; + httpPost(URL); + } + + private void addTestDatasetTDB2(FusekiServer server, String DBname) { + Objects.nonNull(DBname); + addTestDatasetWithName(server, DBname, "TDB2"); + } + + private void checkTask(FusekiServer server, JsonValue v) { + assertNotNull(v); + assertTrue(v.isObject()); + // System.out.println(v); + JsonObject obj = v.getAsObject(); + try { + assertTrue(obj.hasKey("task")); + assertTrue(obj.hasKey("taskId")); + // Not present until it runs : "started" + } catch (AssertionError ex) { + System.out.println(obj); + throw ex; + } + } + + private void checkInTasks(FusekiServer server, String x) { + String url = server.serverURL()+"$/tasks"; + JsonValue v = httpGetJson(url); + assertTrue(v.isArray()); + JsonArray array = v.getAsArray(); + int found = 0; + for ( int i = 0; i < array.size(); i++ ) { + JsonValue jv = array.get(i); + assertTrue(jv.isObject()); + JsonObject obj = jv.getAsObject(); + checkTask(server, obj); + if ( obj.getString("taskId").equals(x) ) { + found++; + } + } + assertEquals(1, found, "Occurrence of taskId count"); + } + + private List<String> runningTasks(FusekiServer server, String... x) { + String url = server.serverURL()+"$/tasks"; + JsonValue v = httpGetJson(url); + assertTrue(v.isArray()); + JsonArray array = v.getAsArray(); + List<String> running = new ArrayList<>(); + for ( int i = 0; i < array.size(); i++ ) { + JsonValue jv = array.get(i); + assertTrue(jv.isObject()); + JsonObject obj = jv.getAsObject(); + if ( isRunning(server, obj) ) + running.add(obj.getString("taskId")); + } + return running; + } + + /** + * Wait for tasks to all finish. + * Algorithm: wait for {@code pause}, then start polling for upto {@code maxWaitMillis}. + * Intervals in milliseconds. + * @param pauseMillis + * @param pollInterval + * @param maxWaitMillis + * @return + */ + private boolean waitForTasksToFinish(FusekiServer server, int pauseMillis, int pollInterval, int maxWaitMillis) { + // Wait for them to finish. + // Divide into chunks + if ( pauseMillis > 0 ) + Lib.sleep(pauseMillis); + long start = System.currentTimeMillis(); + long endTime = start + maxWaitMillis; + final int intervals = maxWaitMillis/pollInterval; + long now = start; + for (int i = 0 ; i < intervals ; i++ ) { + // May have waited (much) longer than the pollInterval : heavily loaded build systems. + if ( now-start > maxWaitMillis ) + break; + List<String> x = runningTasks(server); + if ( x.isEmpty() ) + return true; + Lib.sleep(pollInterval); + now = System.currentTimeMillis(); + } + return false; + } + + private boolean isRunning(FusekiServer server, JsonObject taskObj) { + checkTask(server, taskObj); + return taskObj.hasKey("started") && ! taskObj.hasKey("finished"); + } + + private void askPing(FusekiServer server, String name) { + if ( name.startsWith("/") ) + name = name.substring(1); + try ( TypedInputStream in = httpGet(server.serverURL()+name+"/sparql?query=ASK%7B%7D") ) { + IO.skipToEnd(in); + } + } + + private void adminPing(FusekiServer server, String name) { + try ( TypedInputStream in = httpGet(server.serverURL()+"$/"+opDatasets+"/"+name) ) { + IO.skipToEnd(in); + } + } + + private void checkExists(FusekiServer server, String name) { + adminPing(server, name); + askPing(server, name); + } + + private void checkNotThere(FusekiServer server, String name) { + String n = (name.startsWith("/")) ? name.substring(1) : name; + // Check gone exists. + HttpTest.expect404(()-> adminPing(server, n)); + HttpTest.expect404(() -> askPing(server, n)); + } + + private void checkJsonStatsAll(JsonValue v) { + assertNotNull(v.getAsObject().get("datasets")); + JsonObject a = v.getAsObject().get("datasets").getAsObject(); + for ( String dsname : a.keys() ) { + JsonValue obj = a.get(dsname).getAsObject(); + checkJsonStatsOne(obj); + } + } + + private void checkJsonStatsOne(JsonValue v) { + checkJsonStatsCounters(v); + JsonObject obj1 = v.getAsObject().get("endpoints").getAsObject(); + for ( String srvName : obj1.keys() ) { + JsonObject obj2 = obj1.get(srvName).getAsObject(); + assertTrue(obj2.hasKey("description")); + assertTrue(obj2.hasKey("operation")); + checkJsonStatsCounters(obj2); + } + } + + private void checkJsonStatsCounters(JsonValue v) { + JsonObject obj = v.getAsObject(); + assertTrue(obj.hasKey("Requests")); + assertTrue(obj.hasKey("RequestsGood")); + assertTrue(obj.hasKey("RequestsBad")); + } + + private JsonValue execGetJSON(String url) { + try ( TypedInputStream in = httpGet(url) ) { + assertEqualsContectType(WebContent.contentTypeJSON, in.getContentType()); + return JSON.parse(in); + } + } + + private static void checkJsonDatasetsAll(JsonValue v) { + assertNotNull(v.getAsObject().get("datasets")); + JsonArray a = v.getAsObject().get("datasets").getAsArray(); + for ( JsonValue v2 : a ) + checkJsonDatasetsOne(v2); + } + + private static void checkJsonDatasetsOne(JsonValue v) { + assertTrue(v.isObject()); + JsonObject obj = v.getAsObject(); + assertNotNull(obj.get("ds.name")); + assertNotNull(obj.get("ds.services")); + assertNotNull(obj.get("ds.state")); + assertTrue(obj.get("ds.services").isArray()); + } + + private JsonValue execPostJSON(String url) { + try ( TypedInputStream in = httpPostStream(url, null, null, null) ) { + assertEqualsContectType(WebContent.contentTypeJSON, in.getContentType()); + return JSON.parse(in); + } + } + + /** Expect two string to be non-null and be {@link String#equalsIgnoreCase} */ + private static void assertEqualsContentType(String expected, String actual) { + if ( expected == null && actual == null ) + return; + if ( expected == null || actual == null ) + fail("Expected: "+expected+" Got: "+actual); + if ( ! expected.equalsIgnoreCase(actual) ) + fail("Expected: "+expected+" Got: "+actual); + } +}
jena-fuseki2/jena-fuseki-main/src/test/java/org/apache/jena/fuseki/mod/admin/TestAdmin.java+32 −585 modified@@ -18,140 +18,38 @@ package org.apache.jena.fuseki.mod.admin; -import static org.apache.jena.fuseki.mgt.ServerMgtConst.*; +import static org.apache.jena.fuseki.mgt.ServerMgtConst.opServer; import static org.apache.jena.fuseki.server.ServerConst.opPing; -import static org.apache.jena.fuseki.server.ServerConst.opStats; -import static org.apache.jena.http.HttpOp.*; +import static org.apache.jena.http.HttpOp.httpGet; +import static org.apache.jena.http.HttpOp.httpGetJson; +import static org.apache.jena.http.HttpOp.httpPost; +import static org.apache.jena.http.HttpOp.httpPostRtnJSON; import static org.awaitility.Awaitility.await; -import static org.junit.jupiter.api.Assertions.*; -import static org.junit.jupiter.api.Assumptions.assumeFalse; - -import java.io.FileNotFoundException; -import java.io.IOException; -import java.net.http.HttpRequest.BodyPublisher; -import java.net.http.HttpRequest.BodyPublishers; -import java.nio.file.Path; +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertTrue; +import static org.junit.jupiter.api.Assertions.fail; + import java.util.ArrayList; import java.util.List; -import java.util.Objects; import java.util.concurrent.TimeUnit; -import org.junit.jupiter.api.AfterEach; -import org.junit.jupiter.api.BeforeAll; -import org.junit.jupiter.api.BeforeEach; -import org.junit.jupiter.api.Test; - -import org.apache.commons.lang3.SystemUtils; -import org.apache.jena.atlas.io.IO; -import org.apache.jena.atlas.json.JSON; import org.apache.jena.atlas.json.JsonArray; import org.apache.jena.atlas.json.JsonObject; import org.apache.jena.atlas.json.JsonValue; -import org.apache.jena.atlas.lib.FileOps; import org.apache.jena.atlas.lib.Lib; -import org.apache.jena.atlas.logging.LogCtl; import org.apache.jena.atlas.web.HttpException; -import org.apache.jena.atlas.web.TypedInputStream; -import org.apache.jena.fuseki.Fuseki; -import org.apache.jena.fuseki.ctl.ActionSleep; -import org.apache.jena.fuseki.ctl.JsonConstCtl; -import org.apache.jena.fuseki.main.FusekiServer; -import org.apache.jena.fuseki.main.sys.FusekiModules; -import org.apache.jena.fuseki.mgt.FusekiServerCtl; import org.apache.jena.fuseki.mgt.ServerMgtConst; import org.apache.jena.fuseki.server.ServerConst; -import org.apache.jena.fuseki.system.FusekiLogging; import org.apache.jena.fuseki.test.HttpTest; -import org.apache.jena.riot.WebContent; -import org.apache.jena.sparql.core.DatasetGraph; -import org.apache.jena.sparql.core.DatasetGraphFactory; import org.apache.jena.web.HttpSC; -import org.awaitility.Awaitility; +import org.junit.jupiter.api.Test; /** - * Tests of the admin functionality using a pre-configured dataset - * {@link TestTemplateAddDataset}. + * Tests of the admin functionality using a pre-configured dataset. + * This class does not test adding and deleting of datasets. */ -public class TestAdmin { - - // Name of the dataset in the assembler file. - static String dsTest = "test-ds1"; - static String dsTestInf = "test-ds4"; - - // There are two Fuseki-TDB2 tests: add_delete_dataset_6() and compact_01(). - // - // On certain build systems (GH action/Linux under load, ASF Jenkins sometimes), - // add_delete_dataset_6 fails (transactions active), or compact_01 (gets a 404), - // if the two databases are the same. - static String dsTestTdb2a = "test-tdb2a"; - static String dsTestTdb2b = "test-tdb2b"; - static String fileBase = "testing/Config/"; - - private String serverURL = null; - private FusekiServer server = null; - - @BeforeAll public static void logging() { - FusekiLogging.setLogging(); - } - - @BeforeEach public void startServer() { - System.setProperty("FUSEKI_BASE", "target/run"); - FileOps.clearAll("target/run"); - - server = createServerForTest(); - serverURL = server.serverURL(); - //String adminURL = server.serverURL()+"$"; - //AuthEnv.get().registerUsernamePassword(adminURL, "admin","pw"); - } - - // Exactly the module under test - private static FusekiModules moduleSetup() { - return FusekiModules.create(FMod_Admin.create()); - } - - private FusekiServer createServerForTest() { - FusekiModules modules = moduleSetup(); - DatasetGraph dsg = DatasetGraphFactory.createTxnMem(); - FusekiServer testServer = FusekiServer.create() - .fusekiModules(modules) - .port(0) - .add(datasetName(), dsg) - .addServlet("/$/sleep/*", new ActionSleep()) - .build() - .start(); - return testServer; - } - - @AfterEach public void stopServer() { - if ( server != null ) - server.stop(); - serverURL = null; - FusekiServerCtl.clearUpSystemState(); - } - - protected String urlRoot() { - return serverURL; - } - - protected String datasetName() { - return "dataset"; - } - - protected String datasetPath() { - return "/"+datasetName(); - } - - @BeforeEach public void setLogging() { - LogCtl.setLevel(Fuseki.backupLogName, "ERROR"); - LogCtl.setLevel(Fuseki.compactLogName,"ERROR"); - Awaitility.setDefaultPollDelay(20,TimeUnit.MILLISECONDS); - Awaitility.setDefaultPollInterval(50,TimeUnit.MILLISECONDS); - } - - @AfterEach public void unsetLogging() { - LogCtl.setLevel(Fuseki.backupLogName, "WARN"); - LogCtl.setLevel(Fuseki.compactLogName,"WARN"); - } +public class TestAdmin extends FusekiServerPerTestClass { // --- Ping @@ -178,286 +76,13 @@ protected String datasetPath() { httpPost(urlRoot()+"$/"+opServer); } - // --- List all datasets - - @Test public void list_datasets_1() { - try ( TypedInputStream in = httpGet(urlRoot()+"$/"+opDatasets); ) { - IO.skipToEnd(in); - } - } - - @Test public void list_datasets_2() { - try ( TypedInputStream in = httpGet(urlRoot()+"$/"+opDatasets) ) { - assertEqualsIgnoreCase(WebContent.contentTypeJSON, in.getContentType()); - JsonValue v = JSON.parseAny(in); - assertNotNull(v.getAsObject().get("datasets")); - checkJsonDatasetsAll(v); - } - } - - // Specific dataset - @Test public void list_datasets_3() { - checkExists(datasetName()); - } - - // Specific dataset - @Test public void list_datasets_4() { - HttpTest.expect404( () -> getDatasetDescription("does-not-exist") ); - } - - // Specific dataset - @Test public void list_datasets_5() { - JsonValue v = getDatasetDescription(datasetName()); - checkJsonDatasetsOne(v.getAsObject()); - } - - // Specific dataset - @Test public void add_delete_dataset_1() { - checkNotThere(dsTest); - - addTestDataset(); - - // Check exists. - checkExists(dsTest); - - // Remove it. - deleteDataset(dsTest); - checkNotThere(dsTest); - } - - // Try to add twice - @Test public void add_delete_dataset_2() { - checkNotThere(dsTest); - - try { - Path f = Path.of(fileBase+"config-ds-plain-1.ttl"); - httpPost(urlRoot()+"$/"+opDatasets, - WebContent.contentTypeTurtle+"; charset="+WebContent.charsetUTF8, - BodyPublishers.ofFile(f)); - // Check exists. - checkExists(dsTest); - // Try again. - try { - httpPost(urlRoot()+"$/"+opDatasets, - WebContent.contentTypeTurtle+"; charset="+WebContent.charsetUTF8, - BodyPublishers.ofFile(f)); - } catch (HttpException ex) { - assertEquals(HttpSC.CONFLICT_409, ex.getStatusCode()); - } - } catch (IOException ex) { IO.exception(ex); return; } - - // Check exists. - checkExists(dsTest); - deleteDataset(dsTest); - } - - @Test public void add_delete_dataset_3() { - checkNotThere(dsTest); - addTestDataset(); - checkExists(dsTest); - deleteDataset(dsTest); - checkNotThere(dsTest); - addTestDataset(); - checkExists(dsTest); - deleteDataset(dsTest); - } - - @Test public void add_delete_dataset_4() { - checkNotThere(dsTest); - checkNotThere(dsTestInf); - addTestDatasetInf(); - checkNotThere(dsTest); - checkExists(dsTestInf); - - deleteDataset(dsTestInf); - checkNotThere(dsTestInf); - addTestDatasetInf(); - checkExists(dsTestInf); - deleteDataset(dsTestInf); - } - - @Test public void add_delete_dataset_5() { - // New style operations : cause two fuseki:names - addTestDataset(fileBase+"config-ds-plain-2.ttl"); - checkExists("test-ds2"); - } - - @Test public void add_delete_dataset_6() { - String testDB = dsTestTdb2a; - assumeNotWindows(); - - checkNotThere(testDB); - - addTestDatasetTDB2(testDB); - - // Check exists. - checkExists(testDB); - - // Remove it. - deleteDataset(testDB); - checkNotThere(testDB); - } - - @Test public void add_error_1() { - HttpTest.execWithHttpException(HttpSC.BAD_REQUEST_400, - ()-> addTestDataset(fileBase+"config-ds-bad-name-1.ttl")); - } - - @Test public void add_error_2() { - HttpTest.execWithHttpException(HttpSC.BAD_REQUEST_400, - ()-> addTestDataset(fileBase+"config-ds-bad-name-2.ttl")); - } - - @Test public void add_error_3() { - HttpTest.execWithHttpException(HttpSC.BAD_REQUEST_400, - ()-> addTestDataset(fileBase+"config-ds-bad-name-3.ttl")); - } - - @Test public void add_error_4() { - HttpTest.execWithHttpException(HttpSC.BAD_REQUEST_400, - ()-> addTestDataset(fileBase+"config-ds-bad-name-4.ttl")); - } - - @Test public void delete_dataset_1() { - String name = "NoSuchDataset"; - HttpTest.expect404( ()-> httpDelete(urlRoot()+"$/"+opDatasets+"/"+name) ); - } - - // ---- Backup - - @Test public void create_backup_1() { - String id = null; - try { - JsonValue v = httpPostRtnJSON(urlRoot() + "$/" + opBackup + "/" + datasetName()); - id = v.getAsObject().getString("taskId"); - } finally { - waitForTasksToFinish(1000, 10, 20000); - } - assertNotNull(id); - checkInTasks(id); - - // Check a backup was created - try ( TypedInputStream in = httpGet(urlRoot()+"$/"+opListBackups) ) { - assertEqualsIgnoreCase(WebContent.contentTypeJSON, in.getContentType()); - JsonValue v = JSON.parseAny(in); - assertNotNull(v.getAsObject().get("backups")); - JsonArray a = v.getAsObject().get("backups").getAsArray(); - assertEquals(1, a.size()); - } - - JsonValue task = getTask(id); - assertNotNull(id); - // Expect task success - assertTrue(task.getAsObject().getBoolean(JsonConstCtl.success), "Expected task to be marked as successful"); - } - - @Test - public void create_backup_2() { - HttpTest.expect400(()->{ - JsonValue v = httpPostRtnJSON(urlRoot() + "$/" + opBackup + "/noSuchDataset"); - }); - } - - @Test public void list_backups_1() { - try ( TypedInputStream in = httpGet(urlRoot()+"$/"+opListBackups) ) { - assertEqualsIgnoreCase(WebContent.contentTypeJSON, in.getContentType()); - JsonValue v = JSON.parseAny(in); - assertNotNull(v.getAsObject().get("backups")); - } - } - - // ---- Compact - - @Test public void compact_01() { - assumeNotWindows(); - - String testDB = dsTestTdb2b; - try { - checkNotThere(testDB); - addTestDatasetTDB2(testDB); - checkExists(testDB); - - String id = null; - try { - JsonValue v = httpPostRtnJSON(urlRoot() + "$/" + opCompact + "/" + testDB); - id = v.getAsObject().getString(JsonConstCtl.taskId); - } finally { - waitForTasksToFinish(1000, 500, 20_000); - } - assertNotNull(id); - checkInTasks(id); - - JsonValue task = getTask(id); - // ---- - // The result assertion is throwing NPE occasionally on some heavily loaded CI servers. - // This may be because of server or test code encountering a very long wait. - // These next statements check the assumed structure of the return. - assertNotNull(task, "Task value"); - JsonObject obj = task.getAsObject(); - assertNotNull(obj, "Task.getAsObject()"); - // Provoke code to get a stacktrace. - obj.getBoolean(JsonConstCtl.success); - // ---- - // The assertion we really wanted to check. - // Check task success - assertTrue(task.getAsObject().getBoolean(JsonConstCtl.success), - "Expected task to be marked as successful"); - } finally { - deleteDataset(testDB); - } - } - - @Test public void compact_02() { - HttpTest.expect400(()->{ - JsonValue v = httpPostRtnJSON(urlRoot() + "$/" + opCompact + "/noSuchDataset"); - }); - } - - private void assumeNotWindows() { - assumeFalse(SystemUtils.IS_OS_WINDOWS, "Test may be unstable on Windows due to inability to delete memory-mapped files"); - } - - // ---- Server - - // ---- Stats - - @Test public void stats_1() { - JsonValue v = execGetJSON(urlRoot()+"$/"+opStats); - checkJsonStatsAll(v); - } - - @Test public void stats_2() { - addTestDataset(); - JsonValue v = execGetJSON(urlRoot()+"$/"+opStats+datasetPath()); - checkJsonStatsAll(v); - deleteDataset(dsTest); - } - - @Test public void stats_3() { - addTestDataset(); - HttpTest.expect404(()-> execGetJSON(urlRoot()+"$/"+opStats+"/DoesNotExist")); - deleteDataset(dsTest); - } - - @Test public void stats_4() { - JsonValue v = execPostJSON(urlRoot()+"$/"+opStats); - checkJsonStatsAll(v); - } - - @Test public void stats_5() { - addTestDataset(); - JsonValue v = execPostJSON(urlRoot()+"$/"+opStats+datasetPath()); - checkJsonStatsAll(v); - deleteDataset(dsTest); - } - @Test public void sleep_1() { - String x = execSleepTask(null, 1); + String x = execSleepTask(1); } @Test public void sleep_2() { try { - String x = execSleepTask(null, -1); + String x = execSleepTask(-1); fail("Sleep call unexpectedly succeed"); } catch (HttpException ex) { assertEquals(400, ex.getStatusCode()); @@ -466,7 +91,7 @@ private void assumeNotWindows() { @Test public void sleep_3() { try { - String x = execSleepTask(null, 20*1000+1); + String x = execSleepTask(20*1000+1); fail("Sleep call unexpectedly succeed"); } catch (HttpException ex) { assertEquals(400, ex.getStatusCode()); @@ -476,7 +101,7 @@ private void assumeNotWindows() { // Async task testing @Test public void task_1() { - String x = execSleepTask(null, 10); + String x = execSleepTask(10); assertNotNull(x); Integer.parseInt(x); } @@ -495,7 +120,7 @@ private void assumeNotWindows() { @Test public void task_3() { // Timing dependent. // Create a "long" running task so we can find it. - String x = execSleepTask(null, 100); + String x = execSleepTask(100); checkTask(x); checkInTasks(x); assertNotNull(x); @@ -505,7 +130,7 @@ private void assumeNotWindows() { @Test public void task_4() { // Timing dependent. // Create a "short" running task - String x = execSleepTask(null, 1); + String x = execSleepTask(1); // Check exists in the list of all tasks (should be "finished") checkInTasks(x); String url = urlRoot()+"$/tasks/"+x; @@ -527,31 +152,31 @@ private void assumeNotWindows() { @Test public void task_5() { // Short running task - still in info API call. - String x = execSleepTask(null, 1); + String x = execSleepTask(1); checkInTasks(x); } @Test public void task_6() { - String x1 = execSleepTask(null, 1000); - String x2 = execSleepTask(null, 1000); + String x1 = execSleepTask(1000); + String x2 = execSleepTask(1000); await().timeout(500,TimeUnit.MILLISECONDS).until(() -> runningTasks().size() > 1); await().timeout(2000, TimeUnit.MILLISECONDS).until(() -> runningTasks().isEmpty()); } @Test public void task_7() { try { - String x1 = execSleepTask(null, 1000); - String x2 = execSleepTask(null, 1000); - String x3 = execSleepTask(null, 1000); - String x4 = execSleepTask(null, 1000); + String x1 = execSleepTask(1000); + String x2 = execSleepTask(1000); + String x3 = execSleepTask(1000); + String x4 = execSleepTask(1000); try { // Try to make test more stable on a loaded CI server. // Unloaded the first sleep will fail but due to slowness/burstiness // some tasks above may have completed. - String x5 = execSleepTask(null, 4000); - String x6 = execSleepTask(null, 4000); - String x7 = execSleepTask(null, 4000); - String x8 = execSleepTask(null, 10); + String x5 = execSleepTask(4000); + String x6 = execSleepTask(4000); + String x7 = execSleepTask(4000); + String x8 = execSleepTask(10); fail("Managed to add a 5th test"); } catch (HttpException ex) { assertEquals(HttpSC.BAD_REQUEST_400, ex.getStatusCode()); @@ -561,77 +186,8 @@ private void assumeNotWindows() { } } - /** Expect two string to be non-null and be {@link String#equalsIgnoreCase} */ - private void assertEqualsIgnoreCase(String expected, String actual) { - if ( expected == null && actual == null ) - return; - if ( expected == null || actual == null ) - fail("Expected: "+expected+" Got: "+actual); - if ( ! expected.equalsIgnoreCase(actual) ) - fail("Expected: "+expected+" Got: "+actual); - } - - private JsonValue getTask(String taskId) { - String url = urlRoot()+"$/tasks/"+taskId; - return httpGetJson(url); - } - - private JsonValue getDatasetDescription(String dsName) { - if ( dsName.startsWith("/") ) - dsName = dsName.substring(1); - try (TypedInputStream in = httpGet(urlRoot() + "$/" + opDatasets + "/" + dsName)) { - assertEqualsIgnoreCase(WebContent.contentTypeJSON, in.getContentType()); - JsonValue v = JSON.parse(in); - return v; - } - } - - // -- Add - - private void addTestDataset() { - addTestDataset(fileBase+"config-ds-plain-1.ttl"); - } - - private void addTestDatasetInf() { - addTestDataset(fileBase+"config-ds-inf.ttl"); - } - - private void addTestDatasetTDB2(String DBname) { - Objects.nonNull(DBname); - if ( DBname.equals(dsTestTdb2a) ) { - addTestDataset(fileBase+"config-tdb2a.ttl"); - return; - } - if ( DBname.equals(dsTestTdb2b) ) { - addTestDataset(fileBase+"config-tdb2b.ttl"); - return; - } - throw new IllegalArgumentException("No configuration for "+DBname); - } - - private void addTestDataset(String filename) { - try { - Path f = Path.of(filename); - BodyPublisher body = BodyPublishers.ofFile(f); - String ct = WebContent.contentTypeTurtle; - httpPost(urlRoot()+"$/"+opDatasets, ct, body); - } catch (FileNotFoundException e) { - IO.exception(e); - } - } - - private void deleteDataset(String name) { - httpDelete(urlRoot()+"$/"+opDatasets+"/"+name); - } - - private String execSleepTask(String name, int millis) { + private String execSleepTask(int millis) { String url = urlRoot()+"$/sleep"; - if ( name != null ) { - if ( name.startsWith("/") ) - name = name.substring(1); - url = url + "/"+name; - } - JsonValue v = httpPostRtnJSON(url+"?interval="+millis); String id = v.getAsObject().getString("taskId"); return id; @@ -727,114 +283,5 @@ private boolean isRunning(JsonObject taskObj) { checkTask(taskObj); return taskObj.hasKey("started") && ! taskObj.hasKey("finished"); } - - private void askPing(String name) { - if ( name.startsWith("/") ) - name = name.substring(1); - try ( TypedInputStream in = httpGet(urlRoot()+name+"/sparql?query=ASK%7B%7D") ) { - IO.skipToEnd(in); - } - } - - private void adminPing(String name) { - try ( TypedInputStream in = httpGet(urlRoot()+"$/"+opDatasets+"/"+name) ) { - IO.skipToEnd(in); - } - } - - private void checkExists(String name) { - adminPing(name); - askPing(name); - } - - private void checkExistsNotActive(String name) { - adminPing(name); - try { askPing(name); - fail("askPing did not cause an Http Exception"); - } catch ( HttpException ex ) {} - JsonValue v = getDatasetDescription(name); - assertFalse(v.getAsObject().get("ds.state").getAsBoolean().value()); - } - - private void checkNotThere(String name) { - String n = (name.startsWith("/")) ? name.substring(1) : name; - // Check gone exists. - HttpTest.expect404(()-> adminPing(n) ); - HttpTest.expect404(() -> askPing(n) ); - } - - private void checkJsonDatasetsAll(JsonValue v) { - assertNotNull(v.getAsObject().get("datasets")); - JsonArray a = v.getAsObject().get("datasets").getAsArray(); - for ( JsonValue v2 : a ) - checkJsonDatasetsOne(v2); - } - - private void checkJsonDatasetsOne(JsonValue v) { - assertTrue(v.isObject()); - JsonObject obj = v.getAsObject(); - assertNotNull(obj.get("ds.name")); - assertNotNull(obj.get("ds.services")); - assertNotNull(obj.get("ds.state")); - assertTrue(obj.get("ds.services").isArray()); - } - - private void checkJsonStatsAll(JsonValue v) { - assertNotNull(v.getAsObject().get("datasets")); - JsonObject a = v.getAsObject().get("datasets").getAsObject(); - for ( String dsname : a.keys() ) { - JsonValue obj = a.get(dsname).getAsObject(); - checkJsonStatsOne(obj); - } - } - - private void checkJsonStatsOne(JsonValue v) { - checkJsonStatsCounters(v); - JsonObject obj1 = v.getAsObject().get("endpoints").getAsObject(); - for ( String srvName : obj1.keys() ) { - JsonObject obj2 = obj1.get(srvName).getAsObject(); - assertTrue(obj2.hasKey("description")); - assertTrue(obj2.hasKey("operation")); - checkJsonStatsCounters(obj2); - } - } - - private void checkJsonStatsCounters(JsonValue v) { - JsonObject obj = v.getAsObject(); - assertTrue(obj.hasKey("Requests")); - assertTrue(obj.hasKey("RequestsGood")); - assertTrue(obj.hasKey("RequestsBad")); - } - - private JsonValue execGetJSON(String url) { - try ( TypedInputStream in = httpGet(url) ) { - assertEqualsIgnoreCase(WebContent.contentTypeJSON, in.getContentType()); - return JSON.parse(in); - } - } - - private JsonValue execPostJSON(String url) { - try ( TypedInputStream in = httpPostStream(url, null, null, null) ) { - assertEqualsIgnoreCase(WebContent.contentTypeJSON, in.getContentType()); - return JSON.parse(in); - } - } - - /* - GET /$/ping - POST /$/ping - POST /$/datasets/ - GET /$/datasets/ - DELETE /$/datasets/*{name}* - GET /$/datasets/*{name}* - POST /$/datasets/*{name}*?state=offline - POST /$/datasets/*{name}*?state=active - POST /$/backup/*{name}* - POST /$/compact/*{name}* - GET /$/server - POST /$/server/shutdown - GET /$/stats/ - GET /$/stats/*{name}* - */ }
jena-fuseki2/jena-fuseki-main/src/test/java/org/apache/jena/fuseki/mod/admin/TestTemplateAddDataset.java+0 −183 removed@@ -1,183 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package org.apache.jena.fuseki.mod.admin; - -import static org.junit.jupiter.api.Assertions.assertEquals; -import static org.junit.jupiter.api.Assertions.assertFalse; -import static org.junit.jupiter.api.Assertions.assertThrows; -import static org.junit.jupiter.api.Assertions.assertTrue; - -import java.nio.file.Files; -import java.nio.file.Path; -import java.util.concurrent.TimeUnit; - -import org.junit.jupiter.api.*; - -import org.apache.jena.atlas.lib.FileOps; -import org.apache.jena.atlas.logging.LogCtl; -import org.apache.jena.atlas.web.HttpException; -import org.apache.jena.atlas.web.TypedInputStream; -import org.apache.jena.fuseki.Fuseki; -import org.apache.jena.fuseki.main.FusekiServer; -import org.apache.jena.fuseki.main.sys.FusekiModules; -import org.apache.jena.fuseki.mgt.FusekiServerCtl; -import org.apache.jena.http.HttpOp; -import org.apache.jena.query.QueryExecution; -import org.apache.jena.rdfconnection.RDFConnection; -import org.apache.jena.sparql.core.DatasetGraph; -import org.apache.jena.sparql.core.DatasetGraphFactory; -import org.apache.jena.sparql.exec.http.Params; -import org.apache.jena.web.HttpSC; -import org.awaitility.Awaitility; - -/** - * Tests of the admin functionality on an empty server and using the template mechanism. - * See also {@link TestAdmin}. - */ -public class TestTemplateAddDataset { - - // One server for all tests - private static String serverURL = null; - private static FusekiServer server = null; - - @BeforeAll public static void startServer() { - System.setProperty("FUSEKI_BASE", "target/run"); - FileOps.clearAll("target/run"); - - server = createServerForTest(); - serverURL = server.serverURL(); - } - - // Exactly the module under test - private static FusekiModules moduleSetup() { - return FusekiModules.create(FMod_Admin.create()); - } - - private static FusekiServer createServerForTest() { - FusekiModules modules = moduleSetup(); - DatasetGraph dsg = DatasetGraphFactory.createTxnMem(); - FusekiServer testServer = FusekiServer.create() - .fusekiModules(modules) - .port(0) - .build() - .start(); - return testServer; - } - - @AfterAll public static void stopServer() { - if ( server != null ) - server.stop(); - serverURL = null; - // Clearup FMod_Shiro. - FusekiServerCtl.clearUpSystemState(); - } - - protected String urlRoot() { - return serverURL; - } - - protected String adminURL() { - return serverURL+"$/"; - } - - @BeforeEach public void setLogging() { - LogCtl.setLevel(Fuseki.backupLogName, "ERROR"); - LogCtl.setLevel(Fuseki.compactLogName,"ERROR"); - Awaitility.setDefaultPollDelay(20,TimeUnit.MILLISECONDS); - Awaitility.setDefaultPollInterval(50,TimeUnit.MILLISECONDS); - } - - @AfterEach public void unsetLogging() { - LogCtl.setLevel(Fuseki.backupLogName, "WARN"); - LogCtl.setLevel(Fuseki.compactLogName,"WARN"); - } - - @Order(value = 1) - @Test public void add_delete_api_1() throws Exception { - if ( org.apache.jena.tdb1.sys.SystemTDB.isWindows ) - return; - testAddDelete("db_mem", "mem", false, false); - } - - @Order(value = 2) - @Test public void add_delete_api_2() throws Exception { - if ( org.apache.jena.tdb1.sys.SystemTDB.isWindows ) - return; - // This should fail. - HttpException ex = assertThrows(HttpException.class, ()->testAddDelete("db_mem", "mem", true, false)); - // 409 conflict - "a request conflicts with the current state of the target resource." - // and the target resource is the container "/$/datasets" - assertEquals(HttpSC.CONFLICT_409, ex.getStatusCode()); - } - - private void testAddDelete(String dbName, String dbType, boolean alreadyExists, boolean hasFiles) { - String datasetURL = server.datasetURL(dbName); - Params params = Params.create().add("dbName", dbName).add("dbType", dbType); - - if ( alreadyExists ) - assertTrue(exists(datasetURL)); - else - assertFalse(exists(datasetURL)); - - // Use the template - HttpOp.httpPostForm(adminURL()+"datasets", params); - - RDFConnection conn = RDFConnection.connect(server.datasetURL(dbName)); - conn.update("INSERT DATA { <x:s> <x:p> 123 }"); - int x1 = count(conn); - assertEquals(1, x1); - - Path pathDB = FusekiServerCtl.dirDatabases.resolve(dbName); - - if ( hasFiles ) - assertTrue(Files.exists(pathDB)); - - HttpOp.httpDelete(adminURL()+"datasets/"+dbName); - - assertFalse(exists(datasetURL)); - - //if ( hasFiles ) - assertFalse(Files.exists(pathDB)); - - // Recreate : no contents. - HttpOp.httpPostForm(adminURL()+"datasets", params); - assertTrue(exists(datasetURL), ()->"false: exists("+datasetURL+")"); - int x2 = count(conn); - assertEquals(0, x2); - if ( hasFiles ) - assertTrue(Files.exists(pathDB)); - } - - private static boolean exists(String url) { - try ( TypedInputStream in = HttpOp.httpGet(url) ) { - return true; - } catch (HttpException ex) { - if ( ex.getStatusCode() == HttpSC.NOT_FOUND_404 ) - return false; - throw ex; - } - } - - static int count(RDFConnection conn) { - try ( QueryExecution qExec = conn.query("SELECT (count(*) AS ?C) { ?s ?p ?o }")) { - return qExec.execSelect().next().getLiteral("C").getInt(); - } - } -} -
jena-fuseki2/jena-fuseki-main/src/test/java/org/apache/jena/fuseki/mod/admin/TSMod_Admin.java+33 −0 added@@ -0,0 +1,33 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.jena.fuseki.mod.admin; + +import org.junit.platform.suite.api.SelectClasses; +import org.junit.platform.suite.api.Suite; + +@Suite +@SelectClasses({ + TestAdmin.class, + TestAdminDatabaseOps.class, + TestAdminAddDatasetsConfigFile.class, + TestAdminAddDatasetTemplate.class, + TestFusekiReload.class, +}) + +public class TSMod_Admin {}
jena-fuseki2/jena-fuseki-main/src/test/java/org/apache/jena/fuseki/mod/TS_FusekiMods.java+8 −10 modified@@ -18,27 +18,25 @@ package org.apache.jena.fuseki.mod; -import org.junit.platform.suite.api.SelectClasses; -import org.junit.platform.suite.api.Suite; - -import org.apache.jena.fuseki.mod.admin.TestAdmin; -import org.apache.jena.fuseki.mod.admin.TestFusekiReload; -import org.apache.jena.fuseki.mod.admin.TestTemplateAddDataset; +import org.apache.jena.fuseki.mod.admin.TSMod_Admin; import org.apache.jena.fuseki.mod.metrics.TestModPrometheus; import org.apache.jena.fuseki.mod.shiro.TestModShiro; +import org.junit.platform.suite.api.SelectClasses; +import org.junit.platform.suite.api.Suite; @Suite @SelectClasses({ // Admin - TestAdmin.class, - TestFusekiReload.class, - TestTemplateAddDataset.class, + TSMod_Admin.class, // UI // Prometheus TestModPrometheus.class, - // Apache Shiro + + // Shiro TestModShiro.class, + + // Whole server TestFusekiServer.class }) public class TS_FusekiMods {
jena-fuseki2/jena-fuseki-main/testing/Config/config-tdb2b.ttl+1 −1 modified@@ -15,4 +15,4 @@ PREFIX tdb2: <http://jena.apache.org/2016/tdb#> fuseki:dataset <#dataset> . <#dataset> rdf:type tdb2:DatasetTDB2 ; - tdb2:location "target/tdb2b" . + tdb2:location "--mem--" .
jena-fuseki2/jena-fuseki-main/testing/Config/config-tdb2c.ttl+19 −0 added@@ -0,0 +1,19 @@ +## Licensed under the terms of http://www.apache.org/licenses/LICENSE-2.0 + +PREFIX : <#> +PREFIX fuseki: <http://jena.apache.org/fuseki#> +PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> + +PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#> +PREFIX ja: <http://jena.hpl.hp.com/2005/11/Assembler#> +PREFIX tdb2: <http://jena.apache.org/2016/tdb#> + +<#service1> rdf:type fuseki:Service ; + fuseki:name "test-tdb2b" ; + fuseki:endpoint [ fuseki:name "sparql" ; + fuseki:operation fuseki:query ] ; + fuseki:dataset <#dataset> . + +<#dataset> rdf:type tdb2:DatasetTDB2 ; + # Bad. + tdb2:location "../tdb2c" .
Vulnerability mechanics
Generated on May 9, 2026. Inputs: CWE entries + fix-commit diffs from this CVE's patches. Citations validated against bundle.
References
6- github.com/advisories/GHSA-jq2c-m8gg-mqcmghsaADVISORY
- lists.apache.org/thread/qmm21som8zct813vx6dfd1phnfro6mwqghsavendor-advisoryWEB
- nvd.nist.gov/vuln/detail/CVE-2025-49656ghsaADVISORY
- www.openwall.com/lists/oss-security/2025/07/21/1ghsaWEB
- github.com/apache/jena/commit/03c5265910aa3a27907bf54f6b4aaae3409afa4fghsaWEB
- github.com/apache/jena/commit/35350569b4c1fd432d92e7c92af9597c4400debeghsaWEB
News mentions
0No linked articles in our index yet.