diff --git a/archetypes/dcache-nearline-plugin-archetype/pom.xml b/archetypes/dcache-nearline-plugin-archetype/pom.xml index 3fc46bd9257..63e15e5d67c 100644 --- a/archetypes/dcache-nearline-plugin-archetype/pom.xml +++ b/archetypes/dcache-nearline-plugin-archetype/pom.xml @@ -5,7 +5,7 @@ org.dcache archetypes - 10.2.0-SNAPSHOT + 10.2.12 dcache-nearline-plugin-archetype diff --git a/archetypes/pom.xml b/archetypes/pom.xml index 4c5d7da5752..f5f6dad6476 100644 --- a/archetypes/pom.xml +++ b/archetypes/pom.xml @@ -4,7 +4,7 @@ org.dcache dcache-parent - 10.2.0-SNAPSHOT + 10.2.12 archetypes diff --git a/docs/TheBook/pom.xml b/docs/TheBook/pom.xml index 5014bf8821b..97e330cbb06 100644 --- a/docs/TheBook/pom.xml +++ b/docs/TheBook/pom.xml @@ -4,7 +4,7 @@ org.dcache documentation - 10.2.0-SNAPSHOT + 10.2.12 TheBook diff --git a/docs/TheBook/src/main/markdown/cookbook-debugging.md b/docs/TheBook/src/main/markdown/cookbook-debugging.md index 3ec4a4b7213..93c621ce229 100644 --- a/docs/TheBook/src/main/markdown/cookbook-debugging.md +++ b/docs/TheBook/src/main/markdown/cookbook-debugging.md @@ -4,7 +4,7 @@ This chapter provides an information on how to exam running dCache system, ident ## Java Flight recorder -When debugging an issue on a running system often we need to collect jvm performance stats with `Java flight recorder`. Starting from release 7.2 the Java flight recorder attach listener is enabled by default. Site admins can collect and provide developers with additional information when high CPU load, memory consumption or file descriptor leaks are observed. To enable the `flight recorder` _jcmd_ command is used, which is typically provided as a part of `java-11-openjdk-devel` (on RHEL and clones). +When debugging an issue on a running system often we need to collect jvm performance stats with `Java flight recorder`. Starting from release 7.2 the Java flight recorder attach listener is enabled by default. Site admins can collect and provide developers with additional information when high CPU load, memory consumption or file descriptor leaks are observed. To enable the `flight recorder` _jcmd_ command is used, which is typically provided as a part of `java-17-openjdk-headless` (on RHEL and clones). To control recoding the following subcommands of _jcmd_ available: @@ -111,4 +111,4 @@ Overhead Shared Object Symbol 0.74% [kernel] [k] entry_SYSCALL_64 ``` -More info on usage perf usage at [perf Examples](https://www.brendangregg.com/perf.html) by Brendan Gregg. \ No newline at end of file +More info on usage perf usage at [perf Examples](https://www.brendangregg.com/perf.html) by Brendan Gregg. diff --git a/docs/TheBook/src/main/markdown/dcache-minimal-installation.md b/docs/TheBook/src/main/markdown/dcache-minimal-installation.md index 84fda59d8c2..2a79172ccc8 100644 --- a/docs/TheBook/src/main/markdown/dcache-minimal-installation.md +++ b/docs/TheBook/src/main/markdown/dcache-minimal-installation.md @@ -29,7 +29,7 @@ For a minimal test installation: - At least 500 MiB free disk space - Software: - - OpenJDK 11 + - OpenJDK 17 - Postgres SQL Server 9.5 or later - ZooKeeper version 3.5 (in case of a standalone ZooKeeper installation) @@ -40,11 +40,11 @@ help you with your system specifications. Just contact us: . #### Software: -- OpenJDK 11 (java 11 , and java 17 for dCache staring from version 10.1) +- OpenJDK 17 (or OpenJDK 11 for dCache versions before 10.1) - > yum install java-11-openjdk + > yum install java-17-openjdk > - > dnf install java-11-openjdk-devel + > dnf install java-17-openjdk-devel - ZooKeeper version 3.7 (in case of a standalone ZooKeeper installation) diff --git a/docs/TheBook/src/main/markdown/install.md b/docs/TheBook/src/main/markdown/install.md index 8f8a57ef007..44c115541c1 100644 --- a/docs/TheBook/src/main/markdown/install.md +++ b/docs/TheBook/src/main/markdown/install.md @@ -74,14 +74,14 @@ rpm -ivh https://www.dcache.org/old/downloads/1.9/repo/##SERIES##/dcache-##VERSI | 1:dcache-##VERSION##-1 ################################# [100%] ``` -For example, `##SERIES##` could be `9.1` and `##VERSION##` could be `9.1.1`. +For example, `##SERIES##` could be `10.1` and `##VERSION##` could be `10.1.1`. ### Installing prerequisite packages First, install OpenJDK and httpd-tools packages. ```console-root -yum install java-11-openjdk-headless httpd-tools +dnf install java-17-openjdk-headless httpd-tools ``` ### Installing PostgreSQL diff --git a/docs/UserGuide/pom.xml b/docs/UserGuide/pom.xml index a16db92649c..7f4226cdce4 100644 --- a/docs/UserGuide/pom.xml +++ b/docs/UserGuide/pom.xml @@ -4,7 +4,7 @@ org.dcache documentation - 10.2.0-SNAPSHOT + 10.2.12 UserGuide diff --git a/docs/UserGuide/src/main/markdown/macaroons.md b/docs/UserGuide/src/main/markdown/macaroons.md index 3f01557ba00..ab13e47cd2a 100644 --- a/docs/UserGuide/src/main/markdown/macaroons.md +++ b/docs/UserGuide/src/main/markdown/macaroons.md @@ -168,8 +168,8 @@ has a couple of important consequences: dCache defines seven activities that describe what someone using the macaroon is allowed to do. These are `READ_METADATA`, -`UPDATE_METADATA`, `LIST`, `DOWNLOAD`, `MANAGE`, `UPLOAD` and -`DELETE`, +`UPDATE_METADATA`, `LIST`, `DOWNLOAD`, `MANAGE`, `UPLOAD`, +`DELETE` and `STAGE`.
READ_METADATA
@@ -186,7 +186,7 @@ macaroon is allowed to do. These are `READ_METADATA`,

Any request that attempts to update the metadata about the file: modifying the POSIX permissions, updating ACLs, updating - extended attributes, updating QoS or locality (staging).

+ extended attributes, updating QoS.

LIST
@@ -231,6 +231,12 @@ macaroon is allowed to do. These are `READ_METADATA`, inaccessible from one path but accessible from another is namespace management (see MANAGE activity).

+ +
STAGE
+ +
+

Any request that stages (restores) files from tape.

+
The following describes how HTTP requests are mapped to different diff --git a/docs/UserGuide/src/main/markdown/webdav.md b/docs/UserGuide/src/main/markdown/webdav.md index 7b3845592fb..e09c87237dd 100644 --- a/docs/UserGuide/src/main/markdown/webdav.md +++ b/docs/UserGuide/src/main/markdown/webdav.md @@ -1354,7 +1354,8 @@ activities are defined: | DELETE | Delete a file or directory or overwrite an existing file | MANAGE | Rename and move files or directories | READ_METADATA | Obtain file metadata -| UPDATE_METADATA | Modify file metadata, stage a file or change its QoS +| UPDATE_METADATA | Modify file metadata, change its QoS +| STAGE | Stage (restore) a file from tape The READ_METADATA is implied if any other activity is specified. For example, the caveat activity:LIST,DOWNLOAD restricts the diff --git a/docs/pom.xml b/docs/pom.xml index f01342a6acf..fc5d9ab9713 100644 --- a/docs/pom.xml +++ b/docs/pom.xml @@ -4,7 +4,7 @@ org.dcache dcache-parent - 10.2.0-SNAPSHOT + 10.2.12 documentation diff --git a/modules/acl-vehicles/pom.xml b/modules/acl-vehicles/pom.xml index d9d3522f2df..010338ea4a6 100644 --- a/modules/acl-vehicles/pom.xml +++ b/modules/acl-vehicles/pom.xml @@ -4,7 +4,7 @@ org.dcache dcache-parent - 10.2.0-SNAPSHOT + 10.2.12 ../../pom.xml diff --git a/modules/acl/pom.xml b/modules/acl/pom.xml index 98da2aafd26..095262a517d 100644 --- a/modules/acl/pom.xml +++ b/modules/acl/pom.xml @@ -4,7 +4,7 @@ org.dcache dcache-parent - 10.2.0-SNAPSHOT + 10.2.12 ../../pom.xml diff --git a/modules/benchmarks/pom.xml b/modules/benchmarks/pom.xml index 40d77ad87fb..2d84c7db2d1 100644 --- a/modules/benchmarks/pom.xml +++ b/modules/benchmarks/pom.xml @@ -5,7 +5,7 @@ org.dcache dcache-parent - 10.2.0-SNAPSHOT + 10.2.12 ../../pom.xml diff --git a/modules/cells/pom.xml b/modules/cells/pom.xml index 703eba89e64..3ac3c1dc2cb 100644 --- a/modules/cells/pom.xml +++ b/modules/cells/pom.xml @@ -4,7 +4,7 @@ org.dcache dcache-parent - 10.2.0-SNAPSHOT + 10.2.12 ../../pom.xml diff --git a/modules/cells/src/main/java/dmg/cells/network/LocationManagerConnector.java b/modules/cells/src/main/java/dmg/cells/network/LocationManagerConnector.java index 1c299df5074..b1a200b272b 100644 --- a/modules/cells/src/main/java/dmg/cells/network/LocationManagerConnector.java +++ b/modules/cells/src/main/java/dmg/cells/network/LocationManagerConnector.java @@ -54,8 +54,8 @@ public LocationManagerConnector(String cellName, String args, SocketFactory sock @Override protected void started() { _thread = getNucleus().newThread(this, "TunnelConnector-" + _domain); - _thread.start(); _isRunning = true; + _thread.start(); } private StreamEngine connect() @@ -102,8 +102,8 @@ public void run() { } finally { getNucleus().kill(tunnel.getCellName()); } - } catch (InterruptedIOException | ClosedByInterruptException e) { - throw e; + } catch (InterruptedIOException | InterruptedException | ClosedByInterruptException e) { + _log.warn("Connection to {} ({}) interrupted. Reason: {}", _domain, _address, e.toString()); } catch (ExecutionException | IOException e) { String error = Exceptions.meaningfulMessage(Throwables.getRootCause(e)); _log.warn(AlarmMarkerFactory.getMarker(PredefinedAlarm.LOCATION_MANAGER_FAILURE, @@ -116,11 +116,16 @@ public void run() { _status = "Sleeping"; long sleep = random.nextInt(16000) + 4000; _log.warn("Sleeping {} seconds", sleep / 1000); - Thread.sleep(sleep); + try { + Thread.sleep(sleep); + } catch (InterruptedException e) { + // restore interrupted status + Thread.currentThread().interrupt(); + } } - } catch (InterruptedIOException | InterruptedException | ClosedByInterruptException ignored) { } finally { NDC.pop(); + _thread = null; _status = "Terminated"; } } diff --git a/modules/cells/src/main/java/dmg/cells/services/LocationManager.java b/modules/cells/src/main/java/dmg/cells/services/LocationManager.java index a417ddb9561..653f431c1d3 100644 --- a/modules/cells/src/main/java/dmg/cells/services/LocationManager.java +++ b/modules/cells/src/main/java/dmg/cells/services/LocationManager.java @@ -67,6 +67,7 @@ import org.apache.curator.framework.recipes.cache.NodeCacheListener; import org.apache.curator.framework.recipes.cache.PathChildrenCache; import org.apache.curator.framework.recipes.cache.PathChildrenCacheEvent; +import org.apache.curator.framework.recipes.cache.PathChildrenCacheEvent.Type; import org.apache.curator.utils.CloseableUtils; import org.apache.curator.utils.ZKPaths; import org.apache.zookeeper.CreateMode; @@ -216,7 +217,9 @@ public CoreDomainInfo(byte[] bytes) { switch (entry.getScheme()) { case "tls": case "tcp": - endpoints.add(entry); + if (canResoveHost(entry)) { + endpoints.add(entry); + } break; default: LOGGER.warn("Unknown Scheme for LocationManager Cores: {}", entry); @@ -225,10 +228,30 @@ public CoreDomainInfo(byte[] bytes) { } } catch (IOException ie) { throw new IllegalArgumentException( - "Failed deserializing LocationManager Cores as uri: {}", ie.getCause()); + "Failed deserializing LocationManager Cores as uri", ie); } } + /** + * Check if hostname provided by urc can be resolved. + * @param endpoint core domain endpoint + * @return true it host name can be resolved. Otherwise false. + */ + private boolean canResoveHost(URI endpoint) { + var h = endpoint.getHost(); + if (h == null) { + LOGGER.warn("Ignoring URL without host: {}", endpoint); + return false; + } + try { + var ip = java.net.InetAddress.getByName(h); + } catch (UnknownHostException e) { + LOGGER.warn("Ignoring unknown host: {} : {}", endpoint, e.toString()); + return false; + } + return true; + } + void addCore(String scheme, String host, int port) { switch (scheme) { case "tls": @@ -513,40 +536,46 @@ public void close() { public void reset(Mode mode, State state) { } + private boolean hasNoData(ChildData data) { + return data == null || data.getData() == null || data.getData().length == 0; + } + public void update(PathChildrenCacheEvent event) { LOGGER.info("{}", event); String cell; + if (hasNoData(event.getData())) { + LOGGER.warn("Ignoring empty event {}", event.getType()); + return; + } + + String domain = ZKPaths.getNodeFromPath(event.getData().getPath()); + switch (event.getType()) { case CHILD_REMOVED: - cell = connectors.remove(ZKPaths.getNodeFromPath(event.getData().getPath())); + cell = connectors.remove(domain); if (cell != null) { killConnector(cell); } break; case CHILD_UPDATED: - cell = connectors.remove(ZKPaths.getNodeFromPath(event.getData().getPath())); - if (cell != null) { - killConnector(cell); - } - // fall through case CHILD_ADDED: - //Log if the Core Domain Information received is incompatible with previous + CoreDomainInfo info = infoFromZKEvent(event); - String domain = ZKPaths.getNodeFromPath(event.getData().getPath()); + if (info.endpoints.isEmpty()) { + LOGGER.warn("Ignoring invalid core URI", domain); + break; + } + + if (event.getType() == Type.CHILD_UPDATED) { + cell = connectors.remove(domain); + if (cell != null) { + killConnector(cell); + } + } try { if (shouldConnectTo(domain)) { - cell = connectors.remove(domain); - if (cell != null) { - LOGGER.error( - "About to create tunnel to core domain {}, but to my surprise " - + - "a tunnel called {} already exists. Will kill it. Please contact " - + - "support@dcache.org.", domain, cell); - killConnector(cell); - } cell = connectors.put(domain, startConnector(domain, info)); if (cell != null) { LOGGER.error( diff --git a/modules/chimera/pom.xml b/modules/chimera/pom.xml index 56f28b23ab2..f2e53d3edf6 100644 --- a/modules/chimera/pom.xml +++ b/modules/chimera/pom.xml @@ -4,7 +4,7 @@ org.dcache dcache-parent - 10.2.0-SNAPSHOT + 10.2.12 ../../pom.xml diff --git a/modules/common-cli/pom.xml b/modules/common-cli/pom.xml index a3390ba4d9f..40a4b8f0bb7 100644 --- a/modules/common-cli/pom.xml +++ b/modules/common-cli/pom.xml @@ -4,7 +4,7 @@ org.dcache dcache-parent - 10.2.0-SNAPSHOT + 10.2.12 ../../pom.xml diff --git a/modules/common-security/pom.xml b/modules/common-security/pom.xml index 564ec308c0e..1b4325b439a 100644 --- a/modules/common-security/pom.xml +++ b/modules/common-security/pom.xml @@ -4,7 +4,7 @@ org.dcache dcache-parent - 10.2.0-SNAPSHOT + 10.2.12 ../../pom.xml diff --git a/modules/common/pom.xml b/modules/common/pom.xml index 0b994ef2bff..3c6b407740a 100644 --- a/modules/common/pom.xml +++ b/modules/common/pom.xml @@ -4,7 +4,7 @@ org.dcache dcache-parent - 10.2.0-SNAPSHOT + 10.2.12 ../../pom.xml diff --git a/modules/dcache-bulk/pom.xml b/modules/dcache-bulk/pom.xml index 8c95c12e865..8b7eff8b3e7 100644 --- a/modules/dcache-bulk/pom.xml +++ b/modules/dcache-bulk/pom.xml @@ -4,7 +4,7 @@ org.dcache dcache-parent - 10.2.0-SNAPSHOT + 10.2.12 ../../pom.xml diff --git a/modules/dcache-bulk/src/main/java/org/dcache/services/bulk/activity/BulkActivity.java b/modules/dcache-bulk/src/main/java/org/dcache/services/bulk/activity/BulkActivity.java index 49f82faabe0..c1e5f00a60a 100644 --- a/modules/dcache-bulk/src/main/java/org/dcache/services/bulk/activity/BulkActivity.java +++ b/modules/dcache-bulk/src/main/java/org/dcache/services/bulk/activity/BulkActivity.java @@ -114,7 +114,7 @@ protected BulkActivity(String name, TargetType targetType) { retryPolicy = DEFAULT_RETRY_POLICY; } - public void cancel(BulkRequestTarget target) { + public void cancel(String prefix, BulkRequestTarget target) { target.cancel(); } @@ -173,11 +173,12 @@ public void setDescriptors(Set descriptors) { * * @param rid of the request. * @param tid of the target. + * @param prefix target prefix * @param path of the target on which to perform the activity. * @return future result of the activity. * @throws BulkServiceException */ - public abstract ListenableFuture perform(String rid, long tid, FsPath path, FileAttributes attributes) + public abstract ListenableFuture perform(String rid, long tid, String prefix, FsPath path, FileAttributes attributes) throws BulkServiceException; /** diff --git a/modules/dcache-bulk/src/main/java/org/dcache/services/bulk/activity/plugin/delete/DeleteActivity.java b/modules/dcache-bulk/src/main/java/org/dcache/services/bulk/activity/plugin/delete/DeleteActivity.java index 8c3687326b6..7a7d04eae41 100644 --- a/modules/dcache-bulk/src/main/java/org/dcache/services/bulk/activity/plugin/delete/DeleteActivity.java +++ b/modules/dcache-bulk/src/main/java/org/dcache/services/bulk/activity/plugin/delete/DeleteActivity.java @@ -89,9 +89,10 @@ public DeleteActivity(String name, TargetType targetType) { } @Override - public ListenableFuture perform(String rid, long tid, FsPath path, + public ListenableFuture perform(String rid, long tid, String prefix, FsPath path, FileAttributes attributes) { - PnfsDeleteEntryMessage msg = new PnfsDeleteEntryMessage(path.toString()); + FsPath absolutePath = BulkRequestTarget.computeFsPath(prefix, path.toString()); + PnfsDeleteEntryMessage msg = new PnfsDeleteEntryMessage(absolutePath.toString()); msg.setSubject(subject); if (attributes != null && attributes.getFileType() == FileType.DIR && skipDirs) { msg.setSucceeded(); diff --git a/modules/dcache-bulk/src/main/java/org/dcache/services/bulk/activity/plugin/log/LogTargetActivity.java b/modules/dcache-bulk/src/main/java/org/dcache/services/bulk/activity/plugin/log/LogTargetActivity.java index 330ac0158ba..fb00b84186a 100644 --- a/modules/dcache-bulk/src/main/java/org/dcache/services/bulk/activity/plugin/log/LogTargetActivity.java +++ b/modules/dcache-bulk/src/main/java/org/dcache/services/bulk/activity/plugin/log/LogTargetActivity.java @@ -89,7 +89,7 @@ protected LogTargetActivity(String name, TargetType targetType) { } @Override - public ListenableFuture perform(String ruid, long tid, FsPath path, + public ListenableFuture perform(String ruid, long tid, String prefix, FsPath path, FileAttributes attributes) { long now = System.currentTimeMillis(); BulkRequestTarget t = BulkRequestTargetBuilder.builder(null).activity(this.getName()).id(tid) diff --git a/modules/dcache-bulk/src/main/java/org/dcache/services/bulk/activity/plugin/pin/PinActivity.java b/modules/dcache-bulk/src/main/java/org/dcache/services/bulk/activity/plugin/pin/PinActivity.java index 9c237b97cc0..27ed977e9b0 100644 --- a/modules/dcache-bulk/src/main/java/org/dcache/services/bulk/activity/plugin/pin/PinActivity.java +++ b/modules/dcache-bulk/src/main/java/org/dcache/services/bulk/activity/plugin/pin/PinActivity.java @@ -91,18 +91,18 @@ public PinActivity(String name, TargetType targetType) { super(name, targetType); } - public void cancel(BulkRequestTarget target) { - super.cancel(target); + public void cancel(String prefix, BulkRequestTarget target) { + super.cancel(prefix, target); try { - pinManager.send(unpinMessage(id, target)); + pinManager.send(unpinMessage(id, prefix, target)); } catch (CacheException e) { - target.setErrorObject(new BulkServiceException("unable to fetch pnfsid of target in " + target.setErrorObject(new BulkServiceException("unable to fetch pnfsid of target in " + "order to cancel pinning.", e)); } } @Override - public ListenableFuture perform(String rid, long tid, FsPath target, + public ListenableFuture perform(String rid, long tid, String prefix, FsPath path, FileAttributes attributes) { if (id == null) { id = rid; @@ -110,7 +110,8 @@ public ListenableFuture perform(String rid, long tid, FsPath target, try { if (attributes == null) { - attributes = getAttributes(target); + FsPath absolutePath = BulkRequestTarget.computeFsPath(prefix, path.toString()); + attributes = getAttributes(absolutePath); } checkPinnable(attributes); diff --git a/modules/dcache-bulk/src/main/java/org/dcache/services/bulk/activity/plugin/pin/PinManagerActivity.java b/modules/dcache-bulk/src/main/java/org/dcache/services/bulk/activity/plugin/pin/PinManagerActivity.java index 02019cfd741..c8a95205245 100644 --- a/modules/dcache-bulk/src/main/java/org/dcache/services/bulk/activity/plugin/pin/PinManagerActivity.java +++ b/modules/dcache-bulk/src/main/java/org/dcache/services/bulk/activity/plugin/pin/PinManagerActivity.java @@ -61,6 +61,7 @@ LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING import static com.google.common.util.concurrent.Uninterruptibles.getUninterruptibly; import static diskCacheV111.util.CacheException.INVALID_ARGS; +import static org.dcache.services.bulk.util.BulkRequestTarget.computeFsPath; import static org.dcache.services.bulk.util.BulkRequestTarget.State.SKIPPED; import com.google.common.util.concurrent.Futures; @@ -111,9 +112,6 @@ public void handleCompletion(BulkRequestTarget target, Future future) { reply = getUninterruptibly(future); if (reply.getReturnCode() != 0) { target.setErrorObject(reply.getErrorObject()); - } else if (reply instanceof PinManagerPinMessage - && ((PinManagerPinMessage) reply).getLifetime() == -1L) { - target.setState(SKIPPED); } else { target.setState(State.COMPLETED); } @@ -130,11 +128,12 @@ protected FileAttributes getAttributes(FsPath path) throws CacheException { return pnfsHandler.getFileAttributes(path, MINIMALLY_REQUIRED_ATTRIBUTES); } - protected PinManagerUnpinMessage unpinMessage(String id, BulkRequestTarget target) + protected PinManagerUnpinMessage unpinMessage(String id, String prefix, BulkRequestTarget target) throws CacheException { PnfsId pnfsId = target.getPnfsId(); if (pnfsId == null) { - pnfsId = getAttributes(target.getPath()).getPnfsId(); + FsPath absolutePath = computeFsPath(prefix, target.getPath().toString()); + pnfsId = getAttributes(absolutePath).getPnfsId(); } return unpinMessage(id, pnfsId); } diff --git a/modules/dcache-bulk/src/main/java/org/dcache/services/bulk/activity/plugin/pin/ReleaseActivity.java b/modules/dcache-bulk/src/main/java/org/dcache/services/bulk/activity/plugin/pin/ReleaseActivity.java index be11099b34f..a3e2f157471 100644 --- a/modules/dcache-bulk/src/main/java/org/dcache/services/bulk/activity/plugin/pin/ReleaseActivity.java +++ b/modules/dcache-bulk/src/main/java/org/dcache/services/bulk/activity/plugin/pin/ReleaseActivity.java @@ -60,6 +60,7 @@ LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING package org.dcache.services.bulk.activity.plugin.pin; import static org.dcache.services.bulk.activity.plugin.pin.ReleaseActivityProvider.REQUEST_ID; +import static org.dcache.services.bulk.util.BulkRequestTarget.computeFsPath; import com.google.common.util.concurrent.Futures; import com.google.common.util.concurrent.ListenableFuture; @@ -77,11 +78,12 @@ public ReleaseActivity(String name, TargetType targetType) { } @Override - public ListenableFuture perform(String rid, long tid, FsPath target, + public ListenableFuture perform(String rid, long tid, String prefix, FsPath path, FileAttributes attributes) { try { if (attributes == null) { - attributes = getAttributes(target); + FsPath absolutePath = computeFsPath(prefix, path.toString()); + attributes = getAttributes(absolutePath); } return pinManager.send(unpinMessage(id, attributes.getPnfsId())); } catch (CacheException e) { diff --git a/modules/dcache-bulk/src/main/java/org/dcache/services/bulk/activity/plugin/pin/StageActivity.java b/modules/dcache-bulk/src/main/java/org/dcache/services/bulk/activity/plugin/pin/StageActivity.java index b460b261530..d915ec463c1 100644 --- a/modules/dcache-bulk/src/main/java/org/dcache/services/bulk/activity/plugin/pin/StageActivity.java +++ b/modules/dcache-bulk/src/main/java/org/dcache/services/bulk/activity/plugin/pin/StageActivity.java @@ -102,18 +102,18 @@ public StageActivity(String name, TargetType targetType) { super(name, targetType); } - public void cancel(BulkRequestTarget target) { - super.cancel(target); + public void cancel(String prefix, BulkRequestTarget target) { + super.cancel(prefix, target); try { - pinManager.send(unpinMessage(id, target)); + pinManager.send(unpinMessage(id, prefix, target)); } catch (CacheException e) { target.setErrorObject(new BulkServiceException("unable to fetch pnfsid of target in " - + "order to cancel staging.", e)); + + "order to cancel staging.", e)); } } @Override - public ListenableFuture perform(String rid, long tid, FsPath target, + public ListenableFuture perform(String rid, long tid, String prefix, FsPath path, FileAttributes attributes) { id = rid; @@ -121,7 +121,9 @@ public ListenableFuture perform(String rid, long tid, FsPath target, /* * refetch the attributes because RP is not stored in the bulk database. */ - attributes = getAttributes(target); + + FsPath absolutePath = BulkRequestTarget.computeFsPath(prefix, path.toString()); + attributes = getAttributes(absolutePath); checkStageable(attributes); @@ -129,7 +131,7 @@ public ListenableFuture perform(String rid, long tid, FsPath target, = new PinManagerPinMessage(attributes, getProtocolInfo(), pnfsHandler.getRestriction(), id, - getLifetimeInMillis(target)); + getLifetimeInMillis(path)); message.setSubject(subject); Optional> skipOption = skipIfOnline(attributes, message); diff --git a/modules/dcache-bulk/src/main/java/org/dcache/services/bulk/activity/plugin/pin/UnpinActivity.java b/modules/dcache-bulk/src/main/java/org/dcache/services/bulk/activity/plugin/pin/UnpinActivity.java index db2106b1b95..44a44473851 100644 --- a/modules/dcache-bulk/src/main/java/org/dcache/services/bulk/activity/plugin/pin/UnpinActivity.java +++ b/modules/dcache-bulk/src/main/java/org/dcache/services/bulk/activity/plugin/pin/UnpinActivity.java @@ -60,6 +60,7 @@ LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING package org.dcache.services.bulk.activity.plugin.pin; import static org.dcache.services.bulk.activity.plugin.pin.UnpinActivityProvider.UNPIN_REQUEST_ID; +import static org.dcache.services.bulk.util.BulkRequestTarget.computeFsPath; import com.google.common.util.concurrent.Futures; import com.google.common.util.concurrent.ListenableFuture; @@ -78,11 +79,12 @@ public UnpinActivity(String name, TargetType targetType) { } @Override - public ListenableFuture perform(String rid, long tid, FsPath target, + public ListenableFuture perform(String rid, long tid, String prefix, FsPath path, FileAttributes attributes) { try { if (attributes == null) { - attributes = getAttributes(target); + FsPath absolutePath = computeFsPath(prefix, path.toString()); + attributes = getAttributes(absolutePath); } return pinManager.send(unpinMessage(id, attributes.getPnfsId())); } catch (CacheException e) { diff --git a/modules/dcache-bulk/src/main/java/org/dcache/services/bulk/activity/plugin/qos/UpdateQoSActivity.java b/modules/dcache-bulk/src/main/java/org/dcache/services/bulk/activity/plugin/qos/UpdateQoSActivity.java index e61b169dab0..4df8fb911a3 100644 --- a/modules/dcache-bulk/src/main/java/org/dcache/services/bulk/activity/plugin/qos/UpdateQoSActivity.java +++ b/modules/dcache-bulk/src/main/java/org/dcache/services/bulk/activity/plugin/qos/UpdateQoSActivity.java @@ -112,10 +112,24 @@ public UpdateQoSActivity(String name, TargetType targetType) { } @Override - public synchronized void cancel(BulkRequestTarget target) { + public synchronized void cancel(String prefix, BulkRequestTarget target) { RemoteQoSRequirementsClient client = new RemoteQoSRequirementsClient(); client.setRequirementsService(qosEngine); - PnfsId pnfsId = target.getAttributes().getPnfsId(); + PnfsId pnfsId = null; + if (target.getAttributes() == null) { + FsPath absolutePath = BulkRequestTarget.computeFsPath(prefix, + target.getPath().toString()); + try { + pnfsId = pnfsHandler.getFileAttributes(absolutePath.toString(), + MINIMALLY_REQUIRED_ATTRIBUTES).getPnfsId(); + } catch (CacheException e) { + LOGGER.error("fileQoSRequirementsModifiedCancelled failed: failed to fetch attributes for {} {}.", + target.getPath().toString(), + e.getMessage()); + } + } else { + pnfsId = target.getAttributes().getPnfsId(); + } try { client.fileQoSRequirementsModifiedCancelled(pnfsId, subject); } catch (QoSException e) { @@ -123,19 +137,20 @@ public synchronized void cancel(BulkRequestTarget target) { e.getMessage()); } responseReceiver.cancel(pnfsId.toString()); - super.cancel(target); + super.cancel(prefix, target); } @Override public ListenableFuture perform(String rid, long tid, - FsPath path, FileAttributes attributes) throws BulkServiceException { + String prefix, FsPath path, FileAttributes attributes) throws BulkServiceException { if (targetQos == null && qosPolicy == null) { return Futures.immediateFailedFuture(new IllegalArgumentException("no target qos or policy given.")); } if (attributes == null) { try { - attributes = pnfsHandler.getFileAttributes(path, MINIMALLY_REQUIRED_ATTRIBUTES); + FsPath absolutePath = BulkRequestTarget.computeFsPath(prefix, path.toString()); + attributes = pnfsHandler.getFileAttributes(absolutePath.toString(), MINIMALLY_REQUIRED_ATTRIBUTES); } catch (CacheException e) { throw new BulkServiceException("failed to retrieve file attributes", e); } @@ -232,6 +247,3 @@ public void setNamespaceHandler(PnfsHandler pnfsHandler) { this.pnfsHandler = pnfsHandler; } } - - - diff --git a/modules/dcache-bulk/src/main/java/org/dcache/services/bulk/job/BulkRequestContainerJob.java b/modules/dcache-bulk/src/main/java/org/dcache/services/bulk/job/BulkRequestContainerJob.java index 7317af31a8d..c7d0269ff24 100644 --- a/modules/dcache-bulk/src/main/java/org/dcache/services/bulk/job/BulkRequestContainerJob.java +++ b/modules/dcache-bulk/src/main/java/org/dcache/services/bulk/job/BulkRequestContainerJob.java @@ -120,7 +120,6 @@ LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING import org.dcache.util.list.ListDirectoryHandler; import org.dcache.vehicles.FileAttributes; import org.dcache.vehicles.PnfsGetFileAttributes; -import org.dcache.vehicles.PnfsResolveSymlinksMessage; import org.slf4j.Logger; import org.slf4j.LoggerFactory; @@ -140,18 +139,7 @@ public final class BulkRequestContainerJob static final AtomicLong taskCounter = new AtomicLong(0L); public static FsPath findAbsolutePath(String prefix, String path) { - FsPath absPath = computeFsPath(null, path); - if (prefix == null) { - return absPath; - } - - FsPath pref = FsPath.create(prefix); - - if (!absPath.hasPrefix(pref)) { - absPath = computeFsPath(prefix, path); - } - - return absPath; + return computeFsPath(prefix, path); } /** @@ -235,7 +223,7 @@ enum ContainerState { * proper paths and attributes from listing. */ enum TaskState { - RESOLVE_PATH, FETCH_ATTRIBUTES, HANDLE_TARGET, HANDLE_DIR_TARGET + FETCH_ATTRIBUTES, HANDLE_TARGET, HANDLE_DIR_TARGET } /** @@ -452,7 +440,7 @@ void cancel() { } if (target != null) { - activity.cancel(target); + activity.cancel(targetPrefix, target); LOGGER.debug("{} - target cancelled for task {}.", ruid, seqNo); } @@ -464,9 +452,6 @@ void doInner() { try { checkForRequestCancellation(); switch (state) { - case RESOLVE_PATH: - resolvePath(); - break; case FETCH_ATTRIBUTES: fetchAttributes(); break; @@ -512,43 +497,15 @@ void performSync() throws InterruptedException { } /** - * (1) symlink resolution on initial targets; bypassed for discovered targets. + * (1) retrieval of required file attributes. */ - private void resolvePath() { - LOGGER.debug("{} - resolvePath, resolving {}", ruid, target.getPath()); - PnfsResolveSymlinksMessage message = new PnfsResolveSymlinksMessage( - target.getPath().toString(), null); - ListenableFuture requestFuture = pnfsHandler.requestAsync( - message); - CellStub.addCallback(requestFuture, new AbstractMessageCallback<>() { - @Override - public void success(PnfsResolveSymlinksMessage message) { - LOGGER.debug("{} - resolvePath {}, callback success.", ruid, target.getPath()); - FsPath path = FsPath.create(message.getResolvedPath()); - if (targetPrefix != null && !path.contains(targetPrefix)) { - path = computeFsPath(targetPrefix, path.toString()); - } - LOGGER.debug("{} - resolvePath, resolved path {}", ruid, path); - target.setPath(path); - state = TaskState.FETCH_ATTRIBUTES; - taskFuture = executor.submit(TargetTask.this); - } + private void fetchAttributes() { + FsPath absolutePath = findAbsolutePath(targetPrefix, + target.getPath().toString()); + LOGGER.debug("{} - fetchAttributes for path {}, prefix {}, absolute path {} ", ruid, target.getPath(), targetPrefix, absolutePath); - @Override - public void failure(int rc, Object error) { - LOGGER.error("{} - resolvePath, callback failure for {}.", ruid, target); - storeOrUpdate(CacheExceptionFactory.exceptionOf( - rc, Objects.toString(error, null))); - } - }, callbackExecutor); - } - /** - * (2) retrieval of required file attributes. - */ - private void fetchAttributes() { - LOGGER.debug("{} - fetchAttributes for path {}", ruid, target.getPath()); - PnfsGetFileAttributes message = new PnfsGetFileAttributes(target.getPath().toString(), + PnfsGetFileAttributes message = new PnfsGetFileAttributes(absolutePath.toString(), MINIMALLY_REQUIRED_ATTRIBUTES); ListenableFuture requestFuture = pnfsHandler.requestAsync( message); @@ -573,7 +530,7 @@ public void failure(int rc, Object error) { } /** - * (3b) either recurs on directory or performs activity on file. + * (2b) either recurs on directory or performs activity on file. */ private void handleTarget() throws InterruptedException { LOGGER.debug("{} - handleTarget for {}, path {}.", ruid, target.getActivity(), @@ -611,17 +568,20 @@ private void performActivity(boolean async) throws InterruptedException { FsPath path = target.getPath(); FileAttributes attributes = target.getAttributes(); LOGGER.debug("{} - performActivity {} on {}.", ruid, activity, path); - storeOrUpdate(null); if (hasBeenSpecificallyCancelled(this)) { LOGGER.debug("{} - performActivity hasBeenSpecificallyCancelled for {}.", ruid, - path); + path); remove(); } try { - activityFuture = activity.perform(ruid, id == null ? seqNo : id, path, attributes); + activityFuture = activity.perform(ruid, + id == null ? seqNo : id, + targetPrefix, + path, + attributes); if (async) { activityFuture.addListener(() -> handleCompletion(), callbackExecutor); permitHolder.throttledRelease(); @@ -1077,7 +1037,7 @@ private void processFileTargets() { for (BulkRequestTarget target : requestTargets) { try { checkForRequestCancellation(); - new TargetTask(target, TaskState.RESOLVE_PATH).submitAsync(); + new TargetTask(target, TaskState.FETCH_ATTRIBUTES).submitAsync(); } catch (InterruptedException e) { /* * Cancel most likely called; stop processing. @@ -1128,4 +1088,4 @@ private void update() { signalStateChange(); } -} \ No newline at end of file +} diff --git a/modules/dcache-bulk/src/main/java/org/dcache/services/bulk/manager/ConcurrentRequestManager.java b/modules/dcache-bulk/src/main/java/org/dcache/services/bulk/manager/ConcurrentRequestManager.java index aa7808a04c3..71bba5998ef 100644 --- a/modules/dcache-bulk/src/main/java/org/dcache/services/bulk/manager/ConcurrentRequestManager.java +++ b/modules/dcache-bulk/src/main/java/org/dcache/services/bulk/manager/ConcurrentRequestManager.java @@ -405,6 +405,7 @@ public void shutdown() throws Exception { if (processorFuture != null) { processorFuture.cancel(true); } + processorExecutorService.shutdown(); requestJobs = null; cancelledTargets = null; requestStore.clearCache(); diff --git a/modules/dcache-bulk/src/main/java/org/dcache/services/bulk/util/BulkRequestTarget.java b/modules/dcache-bulk/src/main/java/org/dcache/services/bulk/util/BulkRequestTarget.java index de7d3632be0..2bbbd1b56be 100644 --- a/modules/dcache-bulk/src/main/java/org/dcache/services/bulk/util/BulkRequestTarget.java +++ b/modules/dcache-bulk/src/main/java/org/dcache/services/bulk/util/BulkRequestTarget.java @@ -87,12 +87,15 @@ public final class BulkRequestTarget { RUNNING}; public static FsPath computeFsPath(String prefix, String target) { - if (prefix == null) { - return FsPath.create(FsPath.ROOT + target); - } else { - return FsPath.create( - FsPath.ROOT + (prefix.endsWith("/") ? prefix : prefix + "/") + target); + FsPath absolutePath = FsPath.create(FsPath.ROOT + target); + if (prefix != null) { + FsPath pref = FsPath.create(prefix); + if (!absolutePath.hasPrefix(pref)) { + absolutePath = FsPath.create( + FsPath.ROOT + (prefix.endsWith("/") ? prefix : prefix + "/") + target); + } } + return absolutePath; } public static final FsPath ROOT_REQUEST_PATH = computeFsPath(null, "=request_target="); diff --git a/modules/dcache-bulk/src/main/resources/org/dcache/services/bulk/bulk.xml b/modules/dcache-bulk/src/main/resources/org/dcache/services/bulk/bulk.xml index 3635b42f18a..a1b5749835f 100644 --- a/modules/dcache-bulk/src/main/resources/org/dcache/services/bulk/bulk.xml +++ b/modules/dcache-bulk/src/main/resources/org/dcache/services/bulk/bulk.xml @@ -262,7 +262,7 @@ - + Core of the service which manages the request and target job lifecycle. diff --git a/modules/dcache-chimera/pom.xml b/modules/dcache-chimera/pom.xml index 7bb64e0a99b..9bde197db79 100644 --- a/modules/dcache-chimera/pom.xml +++ b/modules/dcache-chimera/pom.xml @@ -4,7 +4,7 @@ org.dcache dcache-parent - 10.2.0-SNAPSHOT + 10.2.12 ../../pom.xml diff --git a/modules/dcache-dcap/pom.xml b/modules/dcache-dcap/pom.xml index 38c976663d2..ef9085208f9 100644 --- a/modules/dcache-dcap/pom.xml +++ b/modules/dcache-dcap/pom.xml @@ -4,7 +4,7 @@ org.dcache dcache-parent - 10.2.0-SNAPSHOT + 10.2.12 ../../pom.xml diff --git a/modules/dcache-dcap/src/main/java/org/dcache/pool/movers/DCapProtocol_3_nio.java b/modules/dcache-dcap/src/main/java/org/dcache/pool/movers/DCapProtocol_3_nio.java index c4b7c4422ac..97633103d4a 100644 --- a/modules/dcache-dcap/src/main/java/org/dcache/pool/movers/DCapProtocol_3_nio.java +++ b/modules/dcache-dcap/src/main/java/org/dcache/pool/movers/DCapProtocol_3_nio.java @@ -367,7 +367,12 @@ public void runIO(FileAttributes fileAttributes, String errmsg = "WRITE failed : " + (ioException == null ? "IOError" : Exceptions.messageOrClassName(ioException)); int rc; - if (ioException instanceof OutOfDiskException) { + // TODO: To be checked (in production) if the error is an IOException; + // we can check it only by the error message. + if (ioException instanceof OutOfDiskException || + (ioException != null && ioException.getCause().getMessage() + .contains("No space available.")) + ) { _log.debug(errmsg); rc = CacheException.RESOURCE; } else { @@ -535,7 +540,11 @@ public void runIO(FileAttributes fileAttributes, "SEEK_AND_WRITE failed : " + (ioException == null ? "IOError" : Exceptions.messageOrClassName(ioException)); int rc; - if (ioException instanceof OutOfDiskException) { + // TODO: To be checked (in production) if the error is an IOException; + // we can check it only by the error message. + if (ioException instanceof OutOfDiskException || + (ioException != null && ioException.getCause().getMessage() + .contains("No space available."))) { _log.debug(errmsg); rc = CacheException.RESOURCE; } else { @@ -678,10 +687,19 @@ public void runIO(FileAttributes fileAttributes, if (ioException instanceof OutOfDiskException) { throw ioException; } else { - throw new - DiskErrorCacheException( - "Disk I/O Error " + - (ioException != null ? ioException.toString() : "")); + //TODO since the OutOfDiskException is not catched the error ist IOexception + // this is a workaround + if (ioException != null && ioException.getCause().getMessage().contains("No space available.")) { + throw new + OutOfDiskException( + "No space available." + + ioException.toString()); + } else { + throw new + DiskErrorCacheException( + "Disk I/O Error " + + (ioException != null ? ioException.toString() : "")); + } } } else { if (ioException != null && !(ioException instanceof EOFException)) { @@ -922,6 +940,7 @@ private void doTheWrite(RepositoryChannel fileChannel, // clear interrupted state Thread.interrupted(); throw new InterruptedException(ee.getMessage()); + // TODO: In the case of a "no space left" error message, this will not work. } catch (OutOfDiskException e) { _io_ok = false; ioException = e; diff --git a/modules/dcache-frontend/pom.xml b/modules/dcache-frontend/pom.xml index 582709cd1cd..e399cbcb73b 100644 --- a/modules/dcache-frontend/pom.xml +++ b/modules/dcache-frontend/pom.xml @@ -4,7 +4,7 @@ org.dcache dcache-parent - 10.2.0-SNAPSHOT + 10.2.12 ../../pom.xml diff --git a/modules/dcache-frontend/src/main/java/org/dcache/restful/resources/bulk/BulkResources.java b/modules/dcache-frontend/src/main/java/org/dcache/restful/resources/bulk/BulkResources.java index 6a0bf16da2d..40f3b5e0a01 100644 --- a/modules/dcache-frontend/src/main/java/org/dcache/restful/resources/bulk/BulkResources.java +++ b/modules/dcache-frontend/src/main/java/org/dcache/restful/resources/bulk/BulkResources.java @@ -59,6 +59,7 @@ LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING */ package org.dcache.restful.resources.bulk; +import static org.dcache.http.AuthenticationHandler.getLoginAttributes; import static org.dcache.restful.util.HttpServletRequests.getUserRootAwareTargetPrefix; import static org.dcache.restful.util.JSONUtils.newBadRequestException; @@ -67,6 +68,7 @@ LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING import com.google.common.base.Strings; import com.google.gson.Gson; import com.google.gson.JsonParseException; +import diskCacheV111.util.FsPath; import diskCacheV111.util.PnfsHandler; import io.swagger.annotations.Api; import io.swagger.annotations.ApiOperation; @@ -93,6 +95,7 @@ LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING import javax.ws.rs.DELETE; import javax.ws.rs.DefaultValue; import javax.ws.rs.GET; +import javax.ws.rs.ForbiddenException; import javax.ws.rs.NotAuthorizedException; import javax.ws.rs.PATCH; import javax.ws.rs.POST; @@ -103,9 +106,11 @@ LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING import javax.ws.rs.core.Context; import javax.ws.rs.core.MediaType; import javax.ws.rs.core.Response; +import org.dcache.auth.attributes.LoginAttributes; import org.dcache.auth.attributes.Restriction; import org.dcache.auth.attributes.Restrictions; import org.dcache.cells.CellStub; +import org.dcache.http.PathMapper; import org.dcache.restful.util.HandlerBuilders; import org.dcache.restful.util.RequestUser; import org.dcache.restful.util.bulk.BulkServiceCommunicator; @@ -144,6 +149,9 @@ public final class BulkResources { @Inject private BulkServiceCommunicator service; + @Inject + private PathMapper pathMapper; + @Inject @Named("pnfs-stub") private CellStub pnfsmanager; @@ -235,11 +243,14 @@ public Response submit( Subject subject = getSubject(); Restriction restriction = getRestriction(); PnfsHandler handler = HandlerBuilders.unrestrictedPnfsHandler(pnfsmanager); - BulkRequest request = toBulkRequest(requestPayload, this.request, handler); + FsPath userRoot = LoginAttributes.getUserRoot(getLoginAttributes(request)); + FsPath rootPath = pathMapper.effectiveRoot(userRoot, ForbiddenException::new); + BulkRequest request = toBulkRequest(requestPayload, this.request, handler, rootPath); /* * Frontend sets the URL. The backend service provides the UUID. */ + request.setUrlPrefix(this.request.getRequestURL().toString()); BulkRequestMessage message = new BulkRequestMessage(request, restriction); @@ -498,7 +509,7 @@ public static Restriction getRestriction() { * they are defined in the Bulk service as well. */ @VisibleForTesting - static BulkRequest toBulkRequest(String requestPayload, HttpServletRequest httpServletRequest, PnfsHandler handler) { + static BulkRequest toBulkRequest(String requestPayload, HttpServletRequest httpServletRequest, PnfsHandler handler, FsPath rootPath) { if (Strings.emptyToNull(requestPayload) == null) { throw new BadRequestException("empty request payload."); } @@ -531,10 +542,13 @@ static BulkRequest toBulkRequest(String requestPayload, HttpServletRequest httpS string = removeEntry(map, String.class, "target_prefix", "target-prefix", "targetPrefix"); + if (httpServletRequest != null) { - request.setTargetPrefix(getUserRootAwareTargetPrefix(httpServletRequest, string, handler)); + request.setTargetPrefix(getUserRootAwareTargetPrefix(httpServletRequest, + string != null ? string : rootPath.toString(), + handler)); } else { - request.setTargetPrefix(string); + request.setTargetPrefix(string != null ? string : rootPath.toString()); } string = removeEntry(map, String.class, "expand_directories", "expand-directories", diff --git a/modules/dcache-frontend/src/main/java/org/dcache/restful/resources/tape/ArchiveInfoResources.java b/modules/dcache-frontend/src/main/java/org/dcache/restful/resources/tape/ArchiveInfoResources.java index 718c871a58b..314f9ece1ff 100644 --- a/modules/dcache-frontend/src/main/java/org/dcache/restful/resources/tape/ArchiveInfoResources.java +++ b/modules/dcache-frontend/src/main/java/org/dcache/restful/resources/tape/ArchiveInfoResources.java @@ -160,19 +160,17 @@ public List getArchiveInfo( paths = new ArrayList<>(); for (int i = 0; i < len; ++i) { String requestedPath = jsonArray.getString(i); - String dcachePath = rootPath.chroot(requestedPath).toString(); - paths.add(dcachePath); + paths.add(requestedPath); } } catch (JSONException e) { throw newBadRequestException(requestPayload, e); } var archiveInfos = archiveInfoCollector.getInfo(HandlerBuilders.roleAwarePnfsHandler(pnfsManager), - paths); - - archiveInfos.forEach(ai -> - ai.setPath(FsPath.create(ai.getPath()).stripPrefix(rootPath))); - + rootPath.toString(), + paths); return archiveInfos; } + + } diff --git a/modules/dcache-frontend/src/main/java/org/dcache/restful/resources/tape/ReleaseResources.java b/modules/dcache-frontend/src/main/java/org/dcache/restful/resources/tape/ReleaseResources.java index f278bfacb4c..44bd7a816ae 100644 --- a/modules/dcache-frontend/src/main/java/org/dcache/restful/resources/tape/ReleaseResources.java +++ b/modules/dcache-frontend/src/main/java/org/dcache/restful/resources/tape/ReleaseResources.java @@ -159,6 +159,9 @@ public Response release( JSONArray paths; List targetPaths; + FsPath userRoot = LoginAttributes.getUserRoot(getLoginAttributes(request)); + FsPath rootPath = pathMapper.effectiveRoot(userRoot, ForbiddenException::new); + try { JSONObject reqPayload = new JSONObject(requestPayload); paths = reqPayload.getJSONArray("paths"); @@ -170,7 +173,8 @@ public Response release( int len = paths.length(); targetPaths = new ArrayList<>(); for (int i = 0; i < len; ++i) { - targetPaths.add(paths.getString(i)); + String path = paths.getString(i); + targetPaths.add(path); } } catch (JSONException e) { throw newBadRequestException(requestPayload, e); @@ -178,8 +182,6 @@ public Response release( Subject subject = getSubject(); Restriction restriction = getRestriction(); - FsPath userRoot = LoginAttributes.getUserRoot(getLoginAttributes(request)); - FsPath rootPath = pathMapper.effectiveRoot(userRoot, ForbiddenException::new); /* * For WLCG, this is a fire-and-forget request, so it does not need to diff --git a/modules/dcache-frontend/src/main/java/org/dcache/restful/resources/tape/StageResources.java b/modules/dcache-frontend/src/main/java/org/dcache/restful/resources/tape/StageResources.java index 9876209b3e5..8ac882b6144 100644 --- a/modules/dcache-frontend/src/main/java/org/dcache/restful/resources/tape/StageResources.java +++ b/modules/dcache-frontend/src/main/java/org/dcache/restful/resources/tape/StageResources.java @@ -127,6 +127,7 @@ LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING @Api(value = "tape", authorizations = {@Authorization("basicAuth")}) @Path("tape/stage") public final class StageResources { + private static final String STAGE = "STAGE"; @Context @@ -167,6 +168,8 @@ public StageRequestInfo getStageInfo(@ApiParam("The unique id of the request.") @PathParam("id") String id) { Subject subject = getSubject(); Restriction restriction = getRestriction(); + FsPath userRoot = LoginAttributes.getUserRoot(getLoginAttributes(request)); + FsPath rootPath = pathMapper.effectiveRoot(userRoot, ForbiddenException::new); BulkRequestInfo lastInfo = null; List targetInfos = new ArrayList<>(); @@ -218,7 +221,9 @@ public Response cancel( + "does not belong to that stage request, this request will fail.", required = true) String requestPayload) { - JSONObject reqPayload; + FsPath userRoot = LoginAttributes.getUserRoot(getLoginAttributes(request)); + FsPath rootPath = pathMapper.effectiveRoot(userRoot, ForbiddenException::new); + JSONObject reqPayload; JSONArray paths; try { reqPayload = new JSONObject(requestPayload); @@ -233,7 +238,8 @@ public Response cancel( List targetPaths = new ArrayList<>(); int len = paths.length(); for (int i = 0; i < len; ++i) { - targetPaths.add(paths.getString(i)); + String path = paths.getString(i); + targetPaths.add(path); } Subject subject = getSubject(); diff --git a/modules/dcache-frontend/src/main/java/org/dcache/restful/util/wlcg/ArchiveInfoCollector.java b/modules/dcache-frontend/src/main/java/org/dcache/restful/util/wlcg/ArchiveInfoCollector.java index 67ab5298c50..3298156d8f9 100644 --- a/modules/dcache-frontend/src/main/java/org/dcache/restful/util/wlcg/ArchiveInfoCollector.java +++ b/modules/dcache-frontend/src/main/java/org/dcache/restful/util/wlcg/ArchiveInfoCollector.java @@ -64,6 +64,7 @@ LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING import com.google.common.base.Throwables; import diskCacheV111.util.CacheException; import diskCacheV111.util.FileLocality; +import diskCacheV111.util.FsPath; import diskCacheV111.util.PnfsHandler; import dmg.cells.nucleus.CellCommandListener; import dmg.util.command.Argument; @@ -117,12 +118,12 @@ public String call() throws Exception { private ExecutorService service; private int maxPaths; - public List getInfo(PnfsHandler pnfsHandler, List paths) { + public List getInfo(PnfsHandler pnfsHandler, String prefix, List paths) { Map> futures = new HashMap<>(); List infoList = new ArrayList<>(); for (String path : paths) { - futures.put(path, service.submit(() -> getInfo(path, pnfsHandler))); + futures.put(path, service.submit(() -> getInfo(path, prefix, pnfsHandler))); } for (Entry> future : futures.entrySet()) { @@ -162,9 +163,23 @@ public void setService(ExecutorService service) { this.service = service; } - private FileLocality getInfo(String path, PnfsHandler pnfsHandler) throws CacheException { - FileAttributes attributes = pnfsHandler.getFileAttributes(path, REQUIRED_ATTRIBUTES, + private FileLocality getInfo(String path, String prefix, PnfsHandler pnfsHandler) throws CacheException { + String absolutePath = computeFsPath(prefix, path).toString(); + FileAttributes attributes = pnfsHandler.getFileAttributes(absolutePath, REQUIRED_ATTRIBUTES, ACCESS_MASK, false); return poolMonitor.getFileLocality(attributes, "localhost"); } + + public static FsPath computeFsPath(String prefix, String target) { + FsPath absolutePath = FsPath.create(FsPath.ROOT + target); + if (prefix != null) { + FsPath pref = FsPath.create(prefix); + if (!absolutePath.hasPrefix(pref)) { + absolutePath = FsPath.create( + FsPath.ROOT + (prefix.endsWith("/") ? prefix : prefix + "/") + target); + } + } + return absolutePath; + } + } diff --git a/modules/dcache-frontend/src/test/java/org/dcache/restful/resources/bulk/BulkRequestJsonParseTest.java b/modules/dcache-frontend/src/test/java/org/dcache/restful/resources/bulk/BulkRequestJsonParseTest.java index 33e78a20f3f..fb8d8af9ea7 100644 --- a/modules/dcache-frontend/src/test/java/org/dcache/restful/resources/bulk/BulkRequestJsonParseTest.java +++ b/modules/dcache-frontend/src/test/java/org/dcache/restful/resources/bulk/BulkRequestJsonParseTest.java @@ -59,6 +59,7 @@ LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING */ package org.dcache.restful.resources.bulk; +import static diskCacheV111.util.FsPath.create; import static org.dcache.restful.resources.bulk.BulkResources.toBulkRequest; import static org.junit.Assert.assertEquals; @@ -182,6 +183,6 @@ private void givenJsonWithArrayTargetAttribute() { } private void whenParsed() { - bulkRequest = toBulkRequest(requestJson, null, handler); + bulkRequest = toBulkRequest(requestJson, null, handler, create("/")); } } diff --git a/modules/dcache-ftp/pom.xml b/modules/dcache-ftp/pom.xml index bf466fa01c7..01dcc42e07e 100644 --- a/modules/dcache-ftp/pom.xml +++ b/modules/dcache-ftp/pom.xml @@ -4,7 +4,7 @@ org.dcache dcache-parent - 10.2.0-SNAPSHOT + 10.2.12 ../../pom.xml diff --git a/modules/dcache-gplazma/pom.xml b/modules/dcache-gplazma/pom.xml index ebb45e0549f..378db2a99c5 100644 --- a/modules/dcache-gplazma/pom.xml +++ b/modules/dcache-gplazma/pom.xml @@ -4,7 +4,7 @@ org.dcache dcache-parent - 10.2.0-SNAPSHOT + 10.2.12 ../../pom.xml diff --git a/modules/dcache-history/pom.xml b/modules/dcache-history/pom.xml index 41d4e12ec1d..d4072619129 100644 --- a/modules/dcache-history/pom.xml +++ b/modules/dcache-history/pom.xml @@ -4,7 +4,7 @@ org.dcache dcache-parent - 10.2.0-SNAPSHOT + 10.2.12 ../../pom.xml diff --git a/modules/dcache-info/pom.xml b/modules/dcache-info/pom.xml index 0ea49488ffe..90469b5a9b5 100644 --- a/modules/dcache-info/pom.xml +++ b/modules/dcache-info/pom.xml @@ -4,7 +4,7 @@ org.dcache dcache-parent - 10.2.0-SNAPSHOT + 10.2.12 ../../pom.xml diff --git a/modules/dcache-nearline-spi/pom.xml b/modules/dcache-nearline-spi/pom.xml index 12c6560b3ec..b00bd389476 100644 --- a/modules/dcache-nearline-spi/pom.xml +++ b/modules/dcache-nearline-spi/pom.xml @@ -4,7 +4,7 @@ org.dcache dcache-parent - 10.2.0-SNAPSHOT + 10.2.12 ../../pom.xml diff --git a/modules/dcache-nfs/pom.xml b/modules/dcache-nfs/pom.xml index 72a9ff65901..967f9297465 100644 --- a/modules/dcache-nfs/pom.xml +++ b/modules/dcache-nfs/pom.xml @@ -4,7 +4,7 @@ org.dcache dcache-parent - 10.2.0-SNAPSHOT + 10.2.12 ../../pom.xml diff --git a/modules/dcache-qos/pom.xml b/modules/dcache-qos/pom.xml index 21754ba7afc..778c1343ada 100644 --- a/modules/dcache-qos/pom.xml +++ b/modules/dcache-qos/pom.xml @@ -4,7 +4,7 @@ org.dcache dcache-parent - 10.2.0-SNAPSHOT + 10.2.12 ../../pom.xml diff --git a/modules/dcache-qos/src/main/java/org/dcache/qos/services/engine/provider/PolicyBasedQoSProvider.java b/modules/dcache-qos/src/main/java/org/dcache/qos/services/engine/provider/PolicyBasedQoSProvider.java index ca53ee80a6f..2bf9b7cad95 100644 --- a/modules/dcache-qos/src/main/java/org/dcache/qos/services/engine/provider/PolicyBasedQoSProvider.java +++ b/modules/dcache-qos/src/main/java/org/dcache/qos/services/engine/provider/PolicyBasedQoSProvider.java @@ -59,10 +59,7 @@ LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING */ package org.dcache.qos.services.engine.provider; -import static org.dcache.qos.util.QoSPermissionUtils.canModifyQos; - import diskCacheV111.util.CacheException; -import diskCacheV111.util.PermissionDeniedCacheException; import diskCacheV111.util.PnfsId; import java.util.Collections; import java.util.EnumSet; @@ -71,6 +68,7 @@ LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING import java.util.Optional; import java.util.Set; import javax.security.auth.Subject; + import org.dcache.namespace.FileAttribute; import org.dcache.poolmanager.SerializablePoolMonitor; import org.dcache.qos.QoSDiskSpecification; @@ -115,17 +113,6 @@ public FileQoSRequirements fetchRequirements(FileQoSUpdate update) throws QoSExc return null; } - FileAttributes attributes = descriptor.getAttributes(); - if (attributes.isDefined(FileAttribute.QOS_POLICY) && attributes.getQosPolicy() == null) { - /* - * This is a lazily discovered change, so - * as a matter of consistency it calls for removal - * of the pnfsid from the engine's tracking tables. - */ - engineDao.delete(update.getPnfsId()); - return super.fetchRequirements(update, descriptor); - } - return fetchRequirements(update, descriptor); } @@ -133,9 +120,11 @@ public FileQoSRequirements fetchRequirements(FileQoSUpdate update) throws QoSExc public FileQoSRequirements fetchRequirements(FileQoSUpdate update, FileQoSRequirements descriptor) throws QoSException { FileAttributes attributes = descriptor.getAttributes(); - String name = attributes.getQosPolicy(); + + String name = attributes.getQosPolicyIfPresent().orElse(null); if (name == null) { + engineDao.delete(update.getPnfsId()); return super.fetchRequirements(update, descriptor); } @@ -203,25 +192,14 @@ public void handleModifiedRequirements(FileQoSRequirements newRequirements, Subj FileAttributes modifiedAttributes = new FileAttributes(); - if (currentAttributes.isDefined(FileAttribute.QOS_POLICY) - && !newRequirements.hasRequiredQoSPolicy()) { - modifiedAttributes.setQosPolicy(null); // remove the policy from the stored state - modifyRequirements(pnfsId, currentAttributes, modifiedAttributes, newRequirements, - subject); - return; - } - if (newRequirements.hasRequiredQoSPolicy()) { modifiedAttributes.setQosPolicy(newRequirements.getRequiredQoSPolicy()); modifiedAttributes.setQosState(newRequirements.getRequiredQoSStateIndex()); - } - - if (canModifyQos(subject, isEnableRoles(), currentAttributes)) { - pnfsHandler().setFileAttributes(pnfsId, modifiedAttributes); } else { - throw new PermissionDeniedCacheException("User does not have permissions to set " - + "attributes for " + newRequirements.getPnfsId()); + modifiedAttributes.setQosPolicy(null); // remove the policy from the stored state } + + modifyRequirements(pnfsId, currentAttributes, modifiedAttributes, newRequirements, subject); } @Required diff --git a/modules/dcache-qos/src/main/resources/org/dcache/qos/qos-adjuster.xml b/modules/dcache-qos/src/main/resources/org/dcache/qos/qos-adjuster.xml index beab8b448d4..b63015df6e8 100644 --- a/modules/dcache-qos/src/main/resources/org/dcache/qos/qos-adjuster.xml +++ b/modules/dcache-qos/src/main/resources/org/dcache/qos/qos-adjuster.xml @@ -67,7 +67,7 @@ Preserves the QOS session id generated for the task. - + diff --git a/modules/dcache-resilience/pom.xml b/modules/dcache-resilience/pom.xml index a13c74ad053..19ac1a99f3c 100644 --- a/modules/dcache-resilience/pom.xml +++ b/modules/dcache-resilience/pom.xml @@ -4,7 +4,7 @@ org.dcache dcache-parent - 10.2.0-SNAPSHOT + 10.2.12 ../../pom.xml diff --git a/modules/dcache-spacemanager/pom.xml b/modules/dcache-spacemanager/pom.xml index 490dbd8a0af..40a235df10c 100644 --- a/modules/dcache-spacemanager/pom.xml +++ b/modules/dcache-spacemanager/pom.xml @@ -4,7 +4,7 @@ org.dcache dcache-parent - 10.2.0-SNAPSHOT + 10.2.12 ../../pom.xml diff --git a/modules/dcache-srm/pom.xml b/modules/dcache-srm/pom.xml index 6c0ea76a9d8..a654a6e2494 100644 --- a/modules/dcache-srm/pom.xml +++ b/modules/dcache-srm/pom.xml @@ -4,7 +4,7 @@ org.dcache dcache-parent - 10.2.0-SNAPSHOT + 10.2.12 ../../pom.xml diff --git a/modules/dcache-srm/src/main/java/diskCacheV111/srm/dcache/Storage.java b/modules/dcache-srm/src/main/java/diskCacheV111/srm/dcache/Storage.java index bb3132bd30e..05409d0487b 100644 --- a/modules/dcache-srm/src/main/java/diskCacheV111/srm/dcache/Storage.java +++ b/modules/dcache-srm/src/main/java/diskCacheV111/srm/dcache/Storage.java @@ -151,6 +151,7 @@ LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING import java.util.ArrayList; import java.util.Arrays; import java.util.Collection; +import java.util.Collections; import java.util.Comparator; import java.util.EnumSet; import java.util.HashMap; @@ -1784,7 +1785,7 @@ private String performRemoteTransfer(SRMUser srmUser, remoteTURL.toString(), isVerifyRequired(extraInfo), httpHeaders(extraInfo), credential, - Optional.empty()); + Collections.emptyList()); break; case "http": @@ -1792,7 +1793,7 @@ private String performRemoteTransfer(SRMUser srmUser, 1, 1, remoteAddr, remoteTURL.toString(), isVerifyRequired(extraInfo), httpHeaders(extraInfo), - Optional.empty()); + Collections.emptyList()); break; default: diff --git a/modules/dcache-vehicles/pom.xml b/modules/dcache-vehicles/pom.xml index be10a3a0f3f..50997210fa5 100644 --- a/modules/dcache-vehicles/pom.xml +++ b/modules/dcache-vehicles/pom.xml @@ -4,7 +4,7 @@ org.dcache dcache-parent - 10.2.0-SNAPSHOT + 10.2.12 ../../pom.xml diff --git a/modules/dcache-vehicles/src/main/java/diskCacheV111/util/DiskErrorCacheException.java b/modules/dcache-vehicles/src/main/java/diskCacheV111/util/DiskErrorCacheException.java index 2545d2452fe..5429cb86c9a 100644 --- a/modules/dcache-vehicles/src/main/java/diskCacheV111/util/DiskErrorCacheException.java +++ b/modules/dcache-vehicles/src/main/java/diskCacheV111/util/DiskErrorCacheException.java @@ -14,8 +14,13 @@ */ public class DiskErrorCacheException extends CacheException { + private static final long serialVersionUID = -5386946146340646052L; + public enum FileStoreState { + READ_ONLY, FAILED + } + public DiskErrorCacheException(String msg) { super(CacheException.ERROR_IO_DISK, msg); } @@ -23,4 +28,13 @@ public DiskErrorCacheException(String msg) { public DiskErrorCacheException(String message, Throwable cause) { super(CacheException.ERROR_IO_DISK, message, cause); } + + public FileStoreState checkStatus(String message) { + if (message.contains("No space available.")) { + return FileStoreState.READ_ONLY; + } else { + return FileStoreState.FAILED; + } + } + } diff --git a/modules/dcache-vehicles/src/main/java/diskCacheV111/vehicles/HttpProtocolInfo.java b/modules/dcache-vehicles/src/main/java/diskCacheV111/vehicles/HttpProtocolInfo.java index 8e1cd76d557..2dc15a67d23 100644 --- a/modules/dcache-vehicles/src/main/java/diskCacheV111/vehicles/HttpProtocolInfo.java +++ b/modules/dcache-vehicles/src/main/java/diskCacheV111/vehicles/HttpProtocolInfo.java @@ -1,8 +1,13 @@ package diskCacheV111.vehicles; +import java.io.IOException; import java.net.InetSocketAddress; import java.net.URI; -import java.util.Optional; +import java.util.Collections; +import java.util.List; +import java.util.Set; +import javax.annotation.Nonnull; +import javax.annotation.Nullable; import org.dcache.util.ChecksumType; /** @@ -33,6 +38,7 @@ public enum Disposition { private final InetSocketAddress _clientSocketAddress; private long _transferTime; private long _bytesTransferred; + private String _transferTag; /* TODO: Change this to long (but remember backwards compatibility!) */ private int _sessionId; @@ -42,7 +48,10 @@ public enum Disposition { private final String httpDoorDomainName; private final String path; private final URI _location; + @Nullable private final ChecksumType _wantedChecksum; + @Nonnull + private Set _wantedChecksums; private final Disposition _disposition; @@ -55,7 +64,7 @@ public HttpProtocolInfo(String protocol, int major, int minor, String path, URI location) { this(protocol, major, minor, clientSocketAddress, httpDoorCellName, - httpDoorDomainName, path, location, null, null); + httpDoorDomainName, path, location, null, Collections.emptyList()); } public HttpProtocolInfo(String protocol, int major, int minor, @@ -66,9 +75,15 @@ public HttpProtocolInfo(String protocol, int major, int minor, URI location, Disposition disposition) { this(protocol, major, minor, clientSocketAddress, httpDoorCellName, - httpDoorDomainName, path, location, disposition, null); + httpDoorDomainName, path, location, disposition, + Collections.emptyList()); } + /** + * @param wantedChecksums Desired checksum to calculate for this transfer. + * The first checksum is used as a fall-back for old pools that only support + * a single checksum. + */ public HttpProtocolInfo(String protocol, int major, int minor, InetSocketAddress clientSocketAddress, String httpDoorCellName, @@ -76,7 +91,8 @@ public HttpProtocolInfo(String protocol, int major, int minor, String path, URI location, Disposition disposition, - ChecksumType wantedChecksum) { + @Nonnull + List wantedChecksums) { _name = protocol; _minor = minor; _major = major; @@ -86,7 +102,9 @@ public HttpProtocolInfo(String protocol, int major, int minor, this.path = path; _location = location; _disposition = disposition; - _wantedChecksum = wantedChecksum; + _wantedChecksum = wantedChecksums.isEmpty() ? null : wantedChecksums.get(0); + _wantedChecksums = Set.copyOf(wantedChecksums); + _transferTag = ""; } public String getHttpDoorCellName() { @@ -109,8 +127,8 @@ public void setSessionId(int sessionId) { _sessionId = sessionId; } - public Optional getWantedChecksum() { - return Optional.ofNullable(_wantedChecksum); + public Set getWantedChecksums() { + return _wantedChecksums; } // @@ -155,6 +173,15 @@ public long getBytesTransferred() { return _bytesTransferred; } + public void setTransferTag(String tag) { + _transferTag = tag; + } + + @Override + public String getTransferTag() { + return _transferTag; + } + public String toString() { String sb = getVersionString() + ':' + _clientSocketAddress.getAddress().getHostAddress() + @@ -197,6 +224,18 @@ public URI getLocation() { public Disposition getDisposition() { return _disposition != null ? _disposition : Disposition.ATTACHMENT; } + + private void readObject(java.io.ObjectInputStream stream) + throws ClassNotFoundException, IOException { + stream.defaultReadObject(); + + // Handle objects sent from old doors. + if (_wantedChecksums == null) { + _wantedChecksums = _wantedChecksum == null + ? Collections.emptySet() + : Set.of(_wantedChecksum); + } + } } diff --git a/modules/dcache-vehicles/src/main/java/diskCacheV111/vehicles/MoverInfoMessage.java b/modules/dcache-vehicles/src/main/java/diskCacheV111/vehicles/MoverInfoMessage.java index 8dd74776071..d90f38c3d1b 100644 --- a/modules/dcache-vehicles/src/main/java/diskCacheV111/vehicles/MoverInfoMessage.java +++ b/modules/dcache-vehicles/src/main/java/diskCacheV111/vehicles/MoverInfoMessage.java @@ -22,6 +22,8 @@ public class MoverInfoMessage extends PnfsFileInfoMessage { private Duration _readIdle; private Duration _writeActive; private Duration _writeIdle; + private double bytesRead = Double.NaN; + private double bytesWritten = Double.NaN; private static final long serialVersionUID = -7013160118909496211L; private String _transferPath; @@ -140,6 +142,23 @@ public void setLocalEndpoint(InetSocketAddress endpoint) { public Optional getLocalEndpoint() { return Optional.ofNullable(_localEndpoint); } + + public double getBytesRead() { + return bytesRead; + } + + public void setBytesRead(double bytesRead) { + this.bytesRead = bytesRead; + } + + public double getBytesWritten() { + return bytesWritten; + } + + public void setBytesWritten(double bytesWritten) { + this.bytesWritten = bytesWritten; + } + @Override public String toString() { return "MoverInfoMessage{" + diff --git a/modules/dcache-vehicles/src/main/java/diskCacheV111/vehicles/ProtocolInfo.java b/modules/dcache-vehicles/src/main/java/diskCacheV111/vehicles/ProtocolInfo.java index 52d25f0f22a..b5c45166e24 100644 --- a/modules/dcache-vehicles/src/main/java/diskCacheV111/vehicles/ProtocolInfo.java +++ b/modules/dcache-vehicles/src/main/java/diskCacheV111/vehicles/ProtocolInfo.java @@ -22,4 +22,8 @@ public interface ProtocolInfo extends Serializable { int getMajorVersion(); String getVersionString(); + + default String getTransferTag() { + return ""; + } } diff --git a/modules/dcache-vehicles/src/main/java/diskCacheV111/vehicles/RemoteHttpDataTransferProtocolInfo.java b/modules/dcache-vehicles/src/main/java/diskCacheV111/vehicles/RemoteHttpDataTransferProtocolInfo.java index 724cacc519a..faa20ca1905 100644 --- a/modules/dcache-vehicles/src/main/java/diskCacheV111/vehicles/RemoteHttpDataTransferProtocolInfo.java +++ b/modules/dcache-vehicles/src/main/java/diskCacheV111/vehicles/RemoteHttpDataTransferProtocolInfo.java @@ -3,9 +3,14 @@ import static java.util.Objects.requireNonNull; import com.google.common.collect.ImmutableMap; +import java.io.IOException; import java.net.InetSocketAddress; import java.net.URI; -import java.util.Optional; +import java.util.Collections; +import java.util.List; +import java.util.Set; +import javax.annotation.Nonnull; +import javax.annotation.Nullable; import org.dcache.auth.OpenIdCredential; import org.dcache.util.ChecksumType; @@ -24,22 +29,36 @@ public class RemoteHttpDataTransferProtocolInfo implements IpProtocolInfo { private final boolean isVerificationRequired; private final ImmutableMap headers; private final OpenIdCredential openIdCredential; + @Nullable private final ChecksumType desiredChecksum; + @Nonnull + private Set desiredChecksums; private static final long serialVersionUID = 4482469147378465931L; + /** + * @param desiredChecksums Desired checksum to calculate for this transfer. + * The first checksum is used as a fall-back for old pools that only support + * a single checksum. + */ public RemoteHttpDataTransferProtocolInfo(String protocol, int major, int minor, InetSocketAddress addr, String url, boolean isVerificationRequired, ImmutableMap headers, - Optional desiredChecksum) { - this(protocol, minor, major, addr, url, isVerificationRequired, headers, desiredChecksum, - null); + List desiredChecksums) { + this(protocol, minor, major, addr, url, isVerificationRequired, headers, + desiredChecksums, null); } + /** + * @param desiredChecksums Desired checksum to calculate for this transfer. + * The first checksum is used as a fall-back for old pools that only support + * a single checksum. + */ public RemoteHttpDataTransferProtocolInfo(String protocol, int major, int minor, InetSocketAddress addr, String url, boolean isVerificationRequired, ImmutableMap headers, - Optional desiredChecksum, + @Nonnull + List desiredChecksums, OpenIdCredential openIdCredential) { this.name = protocol; this.minor = minor; @@ -49,7 +68,8 @@ public RemoteHttpDataTransferProtocolInfo(String protocol, int major, this.isVerificationRequired = isVerificationRequired; this.headers = requireNonNull(headers); this.openIdCredential = openIdCredential; - this.desiredChecksum = desiredChecksum.orElse(null); + this.desiredChecksum = desiredChecksums.isEmpty() ? null : desiredChecksums.get(0); + this.desiredChecksums = Set.copyOf(desiredChecksums); } public URI getUri() { @@ -102,7 +122,19 @@ public boolean hasTokenCredential() { return openIdCredential != null; } - public Optional getDesiredChecksum() { - return Optional.ofNullable(desiredChecksum); + public Set getDesiredChecksums() { + return desiredChecksums; + } + + private void readObject(java.io.ObjectInputStream stream) + throws ClassNotFoundException, IOException { + stream.defaultReadObject(); + + // Handle objects sent from old doors. + if (desiredChecksums == null) { + desiredChecksums = desiredChecksum == null + ? Collections.emptySet() + : Set.of(desiredChecksum); + } } } diff --git a/modules/dcache-vehicles/src/main/java/diskCacheV111/vehicles/RemoteHttpsDataTransferProtocolInfo.java b/modules/dcache-vehicles/src/main/java/diskCacheV111/vehicles/RemoteHttpsDataTransferProtocolInfo.java index b6795a73eda..e24ec389ee3 100644 --- a/modules/dcache-vehicles/src/main/java/diskCacheV111/vehicles/RemoteHttpsDataTransferProtocolInfo.java +++ b/modules/dcache-vehicles/src/main/java/diskCacheV111/vehicles/RemoteHttpsDataTransferProtocolInfo.java @@ -7,11 +7,12 @@ import java.security.KeyStoreException; import java.security.PrivateKey; import java.security.cert.X509Certificate; -import java.util.Optional; +import java.util.List; import org.dcache.auth.OpenIdCredential; import org.dcache.util.ChecksumType; import static java.util.Objects.requireNonNull; +import javax.annotation.Nonnull; /** * Provides information for HTTP transfer of a file using SSL/TLS encryption. @@ -23,34 +24,53 @@ public class RemoteHttpsDataTransferProtocolInfo extends RemoteHttpDataTransferP private final PrivateKey key; private final X509Certificate[] certChain; + /** + * @param desiredChecksums Desired checksum to calculate for this transfer. + * The first checksum is used as a fall-back for old pools that only support + * a single checksum. + */ public RemoteHttpsDataTransferProtocolInfo(String protocol, int major, int minor, InetSocketAddress addr, String url, boolean isVerificationRequired, ImmutableMap headers, X509Credential credential, - Optional desiredChecksum) { + @Nonnull + List desiredChecksums) { this(protocol, major, minor, addr, url, isVerificationRequired, headers, credential == null ? null : credential.getKey(), credential == null ? null : credential.getCertificateChain(), - desiredChecksum); + desiredChecksums); } + /** + * @param desiredChecksums Desired checksum to calculate for this transfer. + * The first checksum is used as a fall-back for old pools that only support + * a single checksum. + */ public RemoteHttpsDataTransferProtocolInfo(String protocol, int major, int minor, InetSocketAddress addr, String url, boolean isVerificationRequired, ImmutableMap headers, PrivateKey privateKey, X509Certificate[] certificateChain, - Optional desiredChecksum) { - super(protocol, major, minor, addr, url, isVerificationRequired, headers, desiredChecksum); + @Nonnull + List desiredChecksums) { + super(protocol, major, minor, addr, url, isVerificationRequired, headers, + desiredChecksums); key = privateKey; certChain = certificateChain; } + /** + * @param desiredChecksums Desired checksum to calculate for this transfer. + * The first checksum is used as a fall-back for old pools that only support + * a single checksum. + */ public RemoteHttpsDataTransferProtocolInfo(String protocol, int major, int minor, InetSocketAddress addr, String url, boolean isVerificationRequired, ImmutableMap headers, - Optional desiredChecksum, + @Nonnull + List desiredChecksums, OpenIdCredential token) { - super(protocol, major, minor, addr, url, isVerificationRequired, headers, desiredChecksum, - token); + super(protocol, major, minor, addr, url, isVerificationRequired, headers, + desiredChecksums, token); key = null; certChain = null; } diff --git a/modules/dcache-vehicles/src/main/java/org/dcache/util/FileAttributesBuilder.java b/modules/dcache-vehicles/src/main/java/org/dcache/util/FileAttributesBuilder.java index d02e18a8aad..76b6d9976d4 100644 --- a/modules/dcache-vehicles/src/main/java/org/dcache/util/FileAttributesBuilder.java +++ b/modules/dcache-vehicles/src/main/java/org/dcache/util/FileAttributesBuilder.java @@ -1,7 +1,7 @@ /* * dCache - http://www.dcache.org/ * - * Copyright (C) 2021-2022 Deutsches Elektronen-Synchrotron + * Copyright (C) 2021-2024 Deutsches Elektronen-Synchrotron * * This program is free software: you can redistribute it and/or modify * it under the terms of the GNU Affero General Public License as @@ -20,6 +20,8 @@ import diskCacheV111.util.PnfsId; import diskCacheV111.vehicles.StorageInfo; + +import java.util.Arrays; import java.util.Collections; import java.util.HashSet; import java.util.Set; @@ -85,6 +87,11 @@ public FileAttributesBuilder withChecksum(Checksum checksum) { return this; } + public FileAttributesBuilder withLocations(String...locations) { + _attributes.setLocations(Arrays.asList(locations)); + return this; + } + public FileAttributes build() { if (!_checksums.isEmpty()) { _attributes.setChecksums(_checksums); diff --git a/modules/dcache-vehicles/src/main/java/org/dcache/vehicles/FileAttributes.java b/modules/dcache-vehicles/src/main/java/org/dcache/vehicles/FileAttributes.java index a448fb0b892..e523e7b3059 100644 --- a/modules/dcache-vehicles/src/main/java/org/dcache/vehicles/FileAttributes.java +++ b/modules/dcache-vehicles/src/main/java/org/dcache/vehicles/FileAttributes.java @@ -831,7 +831,7 @@ public String toString() { @Nonnull private Optional toOptional(FileAttribute attribute, T value) { - return isDefined(attribute) ? Optional.of(value) : Optional.empty(); + return isDefined(attribute) ? Optional.ofNullable(value) : Optional.empty(); } private void readObject(ObjectInputStream stream) diff --git a/modules/dcache-webdav/pom.xml b/modules/dcache-webdav/pom.xml index 0598a92c73a..985963d16d5 100644 --- a/modules/dcache-webdav/pom.xml +++ b/modules/dcache-webdav/pom.xml @@ -4,7 +4,7 @@ org.dcache dcache-parent - 10.2.0-SNAPSHOT + 10.2.12 ../../pom.xml diff --git a/modules/dcache-webdav/src/main/java/org/dcache/webdav/DcacheResourceFactory.java b/modules/dcache-webdav/src/main/java/org/dcache/webdav/DcacheResourceFactory.java index c9f966f38ad..65925aeaae3 100644 --- a/modules/dcache-webdav/src/main/java/org/dcache/webdav/DcacheResourceFactory.java +++ b/modules/dcache-webdav/src/main/java/org/dcache/webdav/DcacheResourceFactory.java @@ -1683,6 +1683,7 @@ private class HttpTransfer extends RedirectedTransfer { * The original request path that will be passed to pool for fall-back redirect. */ private final String _requestPath; + private String _transferTag = ""; public HttpTransfer(PnfsHandler pnfs, Subject subject, Restriction restriction, FsPath path) throws URISyntaxException { @@ -1693,9 +1694,13 @@ public HttpTransfer(PnfsHandler pnfs, Subject subject, var request = ServletRequest.getRequest(); request.setAttribute(TRANSACTION_ATTRIBUTE, getTransaction()); _requestPath = Requests.stripToPath(request.getRequestURL().toString()); + _transferTag = request.getHeader("SciTag"); } protected ProtocolInfo createProtocolInfo(InetSocketAddress address) { + List wantedChecksums = _wantedChecksum == null + ? Collections.emptyList() + : List.of(_wantedChecksum); HttpProtocolInfo protocolInfo = new HttpProtocolInfo( _isSSL ? PROTOCOL_INFO_SSL_NAME : PROTOCOL_INFO_NAME, @@ -1706,8 +1711,9 @@ protected ProtocolInfo createProtocolInfo(InetSocketAddress address) { _requestPath, _location, _disposition, - _wantedChecksum); + wantedChecksums); protocolInfo.setSessionId((int) getId()); + protocolInfo.setTransferTag(_transferTag); return protocolInfo; } diff --git a/modules/dcache-webdav/src/main/java/org/dcache/webdav/transfer/RemoteTransferHandler.java b/modules/dcache-webdav/src/main/java/org/dcache/webdav/transfer/RemoteTransferHandler.java index fa5b1765433..3e75f221c2c 100644 --- a/modules/dcache-webdav/src/main/java/org/dcache/webdav/transfer/RemoteTransferHandler.java +++ b/modules/dcache-webdav/src/main/java/org/dcache/webdav/transfer/RemoteTransferHandler.java @@ -151,6 +151,7 @@ import static diskCacheV111.services.TransferManagerHandler.RECEIVED_FIRST_POOL_REPLY_STATE; import static dmg.util.CommandException.checkCommand; +import java.util.Collections; import static java.util.stream.Collectors.groupingBy; import static java.util.stream.Collectors.summingInt; @@ -1030,6 +1031,9 @@ private IpProtocolInfo buildProtocolInfo() throws ErrorResponseException { Optional desiredChecksum = _wantDigest.flatMap( Checksums::parseWantDigest); + var desiredChecksums = desiredChecksum + .map(List::of) + .orElseGet(Collections::emptyList); switch (_type) { case GSIFTP: @@ -1042,20 +1046,21 @@ private IpProtocolInfo buildProtocolInfo() throws ErrorResponseException { return new RemoteHttpDataTransferProtocolInfo("RemoteHttpDataTransfer", 1, 1, address, _destination.toASCIIString(), _flags.contains(TransferFlag.REQUIRE_VERIFICATION), - _transferHeaders, desiredChecksum); + _transferHeaders, desiredChecksums); case HTTPS: if (_source == CredentialSource.OIDC) { return new RemoteHttpsDataTransferProtocolInfo("RemoteHttpsDataTransfer", 1, 1, address, _destination.toASCIIString(), _flags.contains(TransferFlag.REQUIRE_VERIFICATION), - _transferHeaders, desiredChecksum, _oidCredential); + _transferHeaders, desiredChecksums, + _oidCredential); } else { return new RemoteHttpsDataTransferProtocolInfo("RemoteHttpsDataTransfer", 1, 1, address, _destination.toASCIIString(), _flags.contains(TransferFlag.REQUIRE_VERIFICATION), _transferHeaders, _privateKey, _certificateChain, - desiredChecksum); + desiredChecksums); } } diff --git a/modules/dcache-xrootd/pom.xml b/modules/dcache-xrootd/pom.xml index cdb95796cfd..fc549cf726d 100644 --- a/modules/dcache-xrootd/pom.xml +++ b/modules/dcache-xrootd/pom.xml @@ -4,7 +4,7 @@ org.dcache dcache-parent - 10.2.0-SNAPSHOT + 10.2.12 ../../pom.xml diff --git a/modules/dcache-xrootd/src/main/java/org/dcache/vehicles/XrootdProtocolInfo.java b/modules/dcache-xrootd/src/main/java/org/dcache/vehicles/XrootdProtocolInfo.java index aec13f61d78..c4063c64e3a 100644 --- a/modules/dcache-xrootd/src/main/java/org/dcache/vehicles/XrootdProtocolInfo.java +++ b/modules/dcache-xrootd/src/main/java/org/dcache/vehicles/XrootdProtocolInfo.java @@ -52,6 +52,8 @@ public enum Flags { private boolean _overwriteAllowed; + private String _transferTag; + public XrootdProtocolInfo(String protocol, int major, int minor, InetSocketAddress clientAddress, CellPath pathToDoor, PnfsId pnfsID, int xrootdFileHandle, UUID uuid, @@ -66,6 +68,7 @@ public XrootdProtocolInfo(String protocol, int major, int minor, _uuid = uuid; _doorAddress = doorAddress; _flags = Sets.newEnumSet(asList(flags), Flags.class); + _transferTag = ""; } public Serializable getDelegatedCredential() { @@ -166,4 +169,13 @@ public void setTpcGid(Long tpcGid) { public void setOverwriteAllowed(boolean overwriteAllowed) { _overwriteAllowed = overwriteAllowed; } + + public void setTransferTag(String tag) { + _transferTag = tag; + } + + @Override + public String getTransferTag() { + return _transferTag; + } } diff --git a/modules/dcache-xrootd/src/main/java/org/dcache/xrootd/door/NettyXrootdServer.java b/modules/dcache-xrootd/src/main/java/org/dcache/xrootd/door/NettyXrootdServer.java index d0be41298ad..5bace62290a 100644 --- a/modules/dcache-xrootd/src/main/java/org/dcache/xrootd/door/NettyXrootdServer.java +++ b/modules/dcache-xrootd/src/main/java/org/dcache/xrootd/door/NettyXrootdServer.java @@ -1,3 +1,4 @@ + package org.dcache.xrootd.door; import static org.dcache.xrootd.plugins.tls.SSLHandlerFactory.SERVER_TLS; @@ -286,7 +287,8 @@ protected void initChannel(Channel ch) throws Exception { } XrootdRedirectHandler handler = new XrootdRedirectHandler(_door, _rootPath, - _requestExecutor, _queryConfig, _appIoQueues); + _requestExecutor, _queryConfig, + _appIoQueues, _expectProxyProtocol); handler.setSigningPolicy(_signingPolicy); handler.setTlsSessionInfo(tlsSessionInfo); pipeline.addLast("redirector", handler); diff --git a/modules/dcache-xrootd/src/main/java/org/dcache/xrootd/door/XrootdRedirectHandler.java b/modules/dcache-xrootd/src/main/java/org/dcache/xrootd/door/XrootdRedirectHandler.java index 3ad10d5aa14..5a3e6ca4ada 100644 --- a/modules/dcache-xrootd/src/main/java/org/dcache/xrootd/door/XrootdRedirectHandler.java +++ b/modules/dcache-xrootd/src/main/java/org/dcache/xrootd/door/XrootdRedirectHandler.java @@ -234,6 +234,7 @@ private Restriction computeRestriction(LoginReply reply) { private final Deque _logins; private final FsPath _rootPath; private final AtomicInteger openRetry = new AtomicInteger(0); + private boolean _expectProxy; /** * Custom entries for kXR_Qconfig requests. @@ -247,9 +248,13 @@ private Restriction computeRestriction(LoginReply reply) { */ private volatile Thread onOpenThread; - public XrootdRedirectHandler(XrootdDoor door, FsPath rootPath, ExecutorService executor, - Map queryConfig, - Map appIoQueues) { + public XrootdRedirectHandler(XrootdDoor door, + FsPath rootPath, + ExecutorService executor, + Map queryConfig, + Map appIoQueues, + boolean expectProxy + ) { super(executor); _door = door; _rootPath = rootPath; @@ -257,6 +262,7 @@ public XrootdRedirectHandler(XrootdDoor door, FsPath rootPath, ExecutorService e _appIoQueues = appIoQueues; _defaultLoginSessionInfo = new LoginSessionInfo(Restrictions.denyAll()); _logins = new ArrayDeque<>(2); + _expectProxy = expectProxy; } @Override @@ -516,7 +522,8 @@ private InetSocketAddress localAddress() { * Use the advertised endpoint, if possble, otherwise fall back to the * address to which the client connected. */ - return _door.publicEndpoint().orElse(getDestinationAddress()); + return _expectProxy ? getDestinationAddress() : + _door.publicEndpoint().orElse(getDestinationAddress()); } /** diff --git a/modules/dcache-xrootd/src/main/java/org/dcache/xrootd/door/XrootdTransfer.java b/modules/dcache-xrootd/src/main/java/org/dcache/xrootd/door/XrootdTransfer.java index 613c35b1555..2a44adaadc0 100644 --- a/modules/dcache-xrootd/src/main/java/org/dcache/xrootd/door/XrootdTransfer.java +++ b/modules/dcache-xrootd/src/main/java/org/dcache/xrootd/door/XrootdTransfer.java @@ -31,12 +31,14 @@ public class XrootdTransfer extends RedirectedTransfer { private boolean proxiedTransfer; private final XrootdTpcInfo tpcInfo; private final Restriction restriction; + private String _transferTag; public XrootdTransfer(PnfsHandler pnfs, Subject subject, - Restriction restriction, FsPath path, Map opaque) throws ParseException { + Restriction restriction, FsPath path, Map opaque) throws ParseException { super(pnfs, subject, restriction, path); this.restriction = requireNonNull(restriction); tpcInfo = new XrootdTpcInfo(opaque); + _transferTag = opaque.getOrDefault("scitag.flow", ""); try { tpcInfo.setUid(Subjects.getUid(subject)); } catch (NoSuchElementException e) { @@ -109,15 +111,18 @@ protected ProtocolInfo getProtocolInfoForPool() { } private XrootdProtocolInfo createXrootdProtocolInfo() { - return new XrootdProtocolInfo(XrootdDoor.XROOTD_PROTOCOL_STRING, - XrootdProtocol.PROTOCOL_VERSION_MAJOR, - XrootdProtocol.PROTOCOL_VERSION_MINOR, - getClientAddress(), - new CellPath(getCellName(), getDomainName()), - getPnfsId(), - _fileHandle, - _uuid, - _doorAddress); + XrootdProtocolInfo protocolInfo = new XrootdProtocolInfo(XrootdDoor.XROOTD_PROTOCOL_STRING, + XrootdProtocol.PROTOCOL_VERSION_MAJOR, + XrootdProtocol.PROTOCOL_VERSION_MINOR, + getClientAddress(), + new CellPath(getCellName(), getDomainName()), + getPnfsId(), + _fileHandle, + _uuid, + _doorAddress); + + protocolInfo.setTransferTag(_transferTag); + return protocolInfo; } @Override diff --git a/modules/dcache/pom.xml b/modules/dcache/pom.xml index 5957306e367..974238430ca 100644 --- a/modules/dcache/pom.xml +++ b/modules/dcache/pom.xml @@ -4,7 +4,7 @@ org.dcache dcache-parent - 10.2.0-SNAPSHOT + 10.2.12 ../../pom.xml diff --git a/modules/dcache/src/main/java/diskCacheV111/doors/CopyManager.java b/modules/dcache/src/main/java/diskCacheV111/doors/CopyManager.java index 129c7c25e1c..d3fb3635e1e 100644 --- a/modules/dcache/src/main/java/diskCacheV111/doors/CopyManager.java +++ b/modules/dcache/src/main/java/diskCacheV111/doors/CopyManager.java @@ -26,6 +26,7 @@ import java.net.InetSocketAddress; import java.net.URI; import java.util.ArrayDeque; +import java.util.Collections; import java.util.Map; import java.util.Optional; import java.util.Queue; @@ -460,7 +461,7 @@ private RemoteHttpDataTransferProtocolInfo createTargetProtocolInfo(String urlRe urlRemote, false, ImmutableMap.of(), - Optional.empty()); + Collections.emptyList()); } diff --git a/modules/dcache/src/main/java/diskCacheV111/poolManager/HttpPoolMgrEngineV3.java b/modules/dcache/src/main/java/diskCacheV111/poolManager/HttpPoolMgrEngineV3.java index 77162b1f014..6291d52bee7 100644 --- a/modules/dcache/src/main/java/diskCacheV111/poolManager/HttpPoolMgrEngineV3.java +++ b/modules/dcache/src/main/java/diskCacheV111/poolManager/HttpPoolMgrEngineV3.java @@ -333,7 +333,7 @@ private void printMenu(PrintWriter pw, String sort, String grep) { "Path", "path" }; pw.println("
"); - pw.println("
"); pw.println(" "); pw.println("
"); @@ -364,6 +364,7 @@ private void printMenu(PrintWriter pw, String sort, String grep) { } + // FIXME: REMOVE! private void printCssFile(PrintWriter pw, String filename) { if (filename.equals("test.html")) { // @@ -394,6 +395,7 @@ private void printCssFile(PrintWriter pw, String filename) { } } + // FIXME: REMOVE! private void printInternalCssFile(PrintWriter pw) { pw.println("body { background-color:orange; }"); pw.println( @@ -548,9 +550,14 @@ private void printConfigurationHeader(PrintWriter pw) { pw.println(""); pw.println(""); + pw.println(""); + pw.println(""); pw.println("
"); - pw.println(""); - } private void printConfigurationPages(PrintWriter pw, String[] urlItems, HttpRequest request) @@ -704,7 +711,7 @@ private void printParameterEntry(PrintWriter pw, Partition p) { int column = 0; int maxColumn = 2; - pw.println("
"); + pw.println("
"); pw.print(""); for (int l = 0; l < (maxColumn + 1); l++) { @@ -991,18 +998,11 @@ private void showRestoreInfo(HTMLWriter html, } private void printPoolManagerHeader(PrintWriter pw, String title) { - pw.println("
"); - pw.println("
"); - pw.println(""); - pw.println("
Birds Home"); - pw.println("		"); if (title != null) { pw.println("

" + title + "

"); } else { pw.println("

Pool Manager Database V3

"); } - pw.println("
"); - } private void showDirectory(PrintWriter pw) { @@ -1241,7 +1241,7 @@ private void queryPool(PrintWriter pw, String poolName) boolean enabled = (Boolean) o[3]; boolean rdOnly = (Boolean) o[5]; long active = (Long) o[4]; - pw.println(""); + pw.println("
"); pw.print(""); @@ -1277,7 +1277,7 @@ private void queryUnit(PrintWriter pw, String unitName) String type = o[1].toString(); Object[] groupList = (Object[]) o[2]; - pw.println("
Enabled : "); pw.print(enabled ? "Yes" : "No"); pw.println("
"); + pw.println("
"); pw.print(""); @@ -1367,8 +1367,8 @@ private void queryLinkList(PrintWriter pw) pw.print(""); pw.print(""); pw.println(""); - int row = 0; + int row = 0; for (LinkProperties lp : list) { pw.println(""); printLinkPropertyRow(pw, lp); @@ -1528,7 +1528,7 @@ private void queryUnitGroup(PrintWriter pw, String groupName) Object[] unitList = (Object[]) o[1]; Object[] linkList = (Object[]) o[2]; - pw.println("
Selection Unit : "); pw.print(unitName); pw.println("
ReadWriteCacheP2p
"); + pw.println("
"); pw.print(""); @@ -1560,7 +1560,7 @@ private void queryPoolGroup(PrintWriter pw, String groupName) Object[] poolList = (Object[]) o[1]; Object[] linkList = (Object[]) o[2]; - pw.println("
Unit Group : "); pw.print(groupName); pw.println("
"); + pw.println("
"); pw.print(""); diff --git a/modules/dcache/src/main/java/diskCacheV111/poolManager/RequestContainerV5.java b/modules/dcache/src/main/java/diskCacheV111/poolManager/RequestContainerV5.java index eea738a91d0..c489b6c511d 100644 --- a/modules/dcache/src/main/java/diskCacheV111/poolManager/RequestContainerV5.java +++ b/modules/dcache/src/main/java/diskCacheV111/poolManager/RequestContainerV5.java @@ -78,6 +78,7 @@ import javax.annotation.Nullable; import javax.annotation.concurrent.GuardedBy; import org.dcache.cells.CellStub; +import org.dcache.namespace.FileAttribute; import org.dcache.poolmanager.CostException; import org.dcache.poolmanager.Partition; import org.dcache.poolmanager.PartitionManager; @@ -363,10 +364,12 @@ public void poolStatusChanged(String poolName, int poolStatus) { * * in this construction we will fall down to next case */ - if (rph.getPoolCandidate().equals(POOL_UNKNOWN_STRING)) { + if (rph.getPoolCandidate().equals(POOL_UNKNOWN_STRING) || rph.expectedOnPool(poolName)) { LOGGER.info("Restore Manager : retrying : {}", rph); rph.retry(); } + + // fall through to retry requests scheduled on that pool case PoolStatusChangedMessage.DOWN: /* * if pool is down, re-try all request scheduled to this @@ -1039,6 +1042,16 @@ public String getPoolCandidate() { } } + /** + * Returns true if file is expected to be on specified pool. + * @param poolName pool name to check. + * @return true if file is expected to be on specified pool. + */ + public boolean expectedOnPool(String poolName) { + return _fileAttributes.isDefined(FileAttribute.LOCATIONS) + && _fileAttributes.getLocations().contains(poolName); + } + private String getPoolCandidateState() { if (_stageCandidate.isPresent()) { return _stageCandidate.get().name(); diff --git a/modules/dcache/src/main/java/diskCacheV111/util/HTMLWriter.java b/modules/dcache/src/main/java/diskCacheV111/util/HTMLWriter.java index b105ea61356..7f97ea9a559 100644 --- a/modules/dcache/src/main/java/diskCacheV111/util/HTMLWriter.java +++ b/modules/dcache/src/main/java/diskCacheV111/util/HTMLWriter.java @@ -70,7 +70,7 @@ public void addHeader(String css, String title) { println(""); println(""); diff --git a/modules/dcache/src/main/java/org/dcache/http/HttpPoolRequestHandler.java b/modules/dcache/src/main/java/org/dcache/http/HttpPoolRequestHandler.java index 3208ee7d039..726d3c2ec2d 100644 --- a/modules/dcache/src/main/java/org/dcache/http/HttpPoolRequestHandler.java +++ b/modules/dcache/src/main/java/org/dcache/http/HttpPoolRequestHandler.java @@ -521,7 +521,7 @@ protected ChannelFuture doOnPut(ChannelHandlerContext context, HttpRequest reque file.truncate(file.getFileAttributes().getSize()); } - file.getProtocolInfo().getWantedChecksum().ifPresent(file::addChecksumType); + file.getProtocolInfo().getWantedChecksums().forEach(file::addChecksumType); _wantedDigest = wantDigest(request).flatMap(Checksums::parseWantDigest); _wantedDigest.ifPresent(file::addChecksumType); diff --git a/modules/dcache/src/main/java/org/dcache/net/FlowMarker.java b/modules/dcache/src/main/java/org/dcache/net/FlowMarker.java index 75003728025..0d7d371214e 100644 --- a/modules/dcache/src/main/java/org/dcache/net/FlowMarker.java +++ b/modules/dcache/src/main/java/org/dcache/net/FlowMarker.java @@ -18,9 +18,13 @@ package org.dcache.net; import com.google.common.net.InetAddresses; + +import diskCacheV111.vehicles.MoverInfoMessage; + import java.net.InetSocketAddress; import java.time.Instant; import java.time.format.DateTimeFormatter; + import org.json.JSONObject; import org.slf4j.Logger; import org.slf4j.LoggerFactory; @@ -40,6 +44,8 @@ public static class FlowMarkerBuilder { private final JSONObject lifecycle = new JSONObject(); private final JSONObject context = new JSONObject(); private final JSONObject flow = new JSONObject(); + private final JSONObject usage = new JSONObject(); + private final JSONObject stats = new JSONObject(); public FlowMarkerBuilder withStartedAt(Instant startTime) { lifecycle.put("start-time", DateTimeFormatter.ISO_INSTANT.format(startTime)); @@ -104,7 +110,32 @@ public FlowMarkerBuilder withProtocol(String proto) { return this; } - public JSONObject build(String state) { + public FlowMarkerBuilder withUsage(double bytesRead, double bytesWritten) { + // approx. - assumes bytesWritten to disk are bytes "received" over the network + // and bytesRead from disk are bytes "sent" to the network + usage.put("received", Double.isFinite(bytesWritten) ? (long) bytesWritten : 0L); + usage.put("sent", Double.isFinite(bytesRead) ? (long) bytesRead : 0L); + return this; + } + + public FlowMarkerBuilder withStats(MoverInfoMessage message) { + stats.put("bytes-transferred", message.getDataTransferred()); + stats.put("connection-time", message.getConnectionTime()); + stats.put("read-bw", Double.isFinite(message.getMeanReadBandwidth()) ? (long) message.getMeanReadBandwidth() : 0L); + stats.put("write-bw", Double.isFinite(message.getMeanWriteBandwidth()) ? (long) message.getMeanWriteBandwidth() : 0L); + stats.put("read-active", message.getReadActive().isPresent() ? message.getReadActive().get().toString() : "0"); + stats.put("read-idle", message.getReadIdle().isPresent() ? message.getReadIdle().get().toString() : "0"); + stats.put("write-active", message.getWriteActive().isPresent() ? message.getWriteActive().get().toString() : "0"); + stats.put("write-idle", message.getWriteIdle().isPresent() ? message.getWriteIdle().get().toString() : "0"); + return this; + } + + public FlowMarkerBuilder withFlowId(String id) { + flow.put("flow-id", id); + return this; + } + + public String build(String state) { switch (state) { case "start": @@ -119,11 +150,19 @@ public JSONObject build(String state) { payload.put("flow-lifecycle", lifecycle); payload.put("context", context); payload.put("flow-id", flow); + payload.put("usage", usage); + if(!stats.isEmpty()) + payload.put("storage-stats", stats); lifecycle.put("state", state); lifecycle.put("current-time", DateTimeFormatter.ISO_INSTANT.format(Instant.now())); - return payload; + String jsonPayload = payload.toString(); + String time = DateTimeFormatter.ISO_INSTANT.format(Instant.now()); + String host = flow.has("src-ip") ? flow.getString("src-ip") : "localhost"; + String syslogHeader = String.format("<134>1 %s %s dCache - firefly-json - ", time, host); + + return syslogHeader + jsonPayload; } } } diff --git a/modules/dcache/src/main/java/org/dcache/pool/classic/DefaultPostTransferService.java b/modules/dcache/src/main/java/org/dcache/pool/classic/DefaultPostTransferService.java index 1b492e5a8ae..bc7c73cbb7b 100644 --- a/modules/dcache/src/main/java/org/dcache/pool/classic/DefaultPostTransferService.java +++ b/modules/dcache/src/main/java/org/dcache/pool/classic/DefaultPostTransferService.java @@ -208,12 +208,14 @@ private static MoverInfoMessage updateIoStatistics(MoverInfoMessage info, SnapshotStatistics writeStats = writes.statistics(); if (readStats.requestedBytes().getN() > 0) { + info.setBytesRead(readStats.transferredBytes().getSum()); info.setMeanReadBandwidth(readStats.instantaneousBandwidth().getMean()); info.setReadActive(reads.active()); info.setReadIdle(reads.idle()); } if (writeStats.requestedBytes().getN() > 0) { + info.setBytesWritten(writeStats.transferredBytes().getSum()); info.setMeanWriteBandwidth(writeStats.instantaneousBandwidth().getMean()); info.setWriteActive(writes.active()); info.setWriteIdle(writes.idle()); @@ -240,8 +242,7 @@ public void sendFinished(Mover mover, MoverInfoMessage moverInfoMessage) { mover.getLocalEndpoint().ifPresent(e -> transferLifeCycle.onEnd(((IpProtocolInfo) mover.getProtocolInfo()).getSocketAddress(), e, - mover.getProtocolInfo(), - mover.getSubject())); + moverInfoMessage)); _door.notify(mover.getPathToDoor(), finished); } diff --git a/modules/dcache/src/main/java/org/dcache/pool/classic/MoverRequestScheduler.java b/modules/dcache/src/main/java/org/dcache/pool/classic/MoverRequestScheduler.java index 7504d09c647..30f678cab77 100644 --- a/modules/dcache/src/main/java/org/dcache/pool/classic/MoverRequestScheduler.java +++ b/modules/dcache/src/main/java/org/dcache/pool/classic/MoverRequestScheduler.java @@ -41,6 +41,7 @@ import org.dcache.pool.movers.Mover; import org.dcache.pool.movers.json.MoverData; import org.dcache.pool.repository.FileStore; +import org.dcache.pool.repository.OutOfDiskException; import org.dcache.util.AdjustableSemaphore; import org.dcache.util.IoPrioritizable; import org.dcache.util.IoPriority; @@ -513,9 +514,27 @@ public void failed(Throwable exc, Void attachment) { .setTransferStatus(CacheException.DEFAULT_ERROR_CODE, "Transfer was killed"); } else if (exc instanceof DiskErrorCacheException) { + FaultAction faultAction = null; + //TODO this is done because the FileStoreState is in another module + // to be improved + switch (((DiskErrorCacheException) exc).checkStatus(exc.getMessage())){ + case READ_ONLY: + faultAction = FaultAction.READONLY; + break; + default: + faultAction = FaultAction.DISABLED; + break; + } FaultEvent faultEvent = new FaultEvent("transfer", - FaultAction.DISABLED, exc.getMessage(), exc); + faultAction, exc.getMessage(), exc); _faultListeners.forEach(l -> l.faultOccurred(faultEvent)); + } else if (exc instanceof OutOfDiskException) { + FaultEvent faultEvent = new FaultEvent( + "post-processing", + FaultAction.READONLY, + exc.getMessage(), exc); + _faultListeners.forEach( + l -> l.faultOccurred(faultEvent)); } postprocess(); } @@ -532,9 +551,25 @@ public void completed(Void result, Void attachment) { @Override public void failed(Throwable exc, Void attachment) { if (exc instanceof DiskErrorCacheException) { + FaultAction faultAction = null; + switch (((DiskErrorCacheException) exc).checkStatus(exc.getMessage())){ + case READ_ONLY: + faultAction = FaultAction.READONLY; + break; + default: + faultAction = FaultAction.DISABLED; + break; + } FaultEvent faultEvent = new FaultEvent( "post-processing", - FaultAction.DISABLED, + faultAction, + exc.getMessage(), exc); + _faultListeners.forEach( + l -> l.faultOccurred(faultEvent)); + } else if (exc instanceof OutOfDiskException) { + FaultEvent faultEvent = new FaultEvent( + "post-processing", + FaultAction.READONLY, exc.getMessage(), exc); _faultListeners.forEach( l -> l.faultOccurred(faultEvent)); diff --git a/modules/dcache/src/main/java/org/dcache/pool/movers/RemoteHttpDataTransferProtocol.java b/modules/dcache/src/main/java/org/dcache/pool/movers/RemoteHttpDataTransferProtocol.java index 8fe1b2949b3..47d5f7eda11 100644 --- a/modules/dcache/src/main/java/org/dcache/pool/movers/RemoteHttpDataTransferProtocol.java +++ b/modules/dcache/src/main/java/org/dcache/pool/movers/RemoteHttpDataTransferProtocol.java @@ -238,7 +238,7 @@ public void runIO(FileAttributes attributes, RepositoryChannel channel, _channel = new MoverChannel<>(access, attributes, info, channel); channel.optionallyAs(ChecksumChannel.class).ifPresent(c -> { - info.getDesiredChecksum().ifPresent(t -> { + info.getDesiredChecksums().forEach(t -> { try { c.addType(t); } catch (IOException e) { diff --git a/modules/dcache/src/main/java/org/dcache/pool/movers/TransferLifeCycle.java b/modules/dcache/src/main/java/org/dcache/pool/movers/TransferLifeCycle.java index 64fc9e6bb9d..2d12803d454 100644 --- a/modules/dcache/src/main/java/org/dcache/pool/movers/TransferLifeCycle.java +++ b/modules/dcache/src/main/java/org/dcache/pool/movers/TransferLifeCycle.java @@ -22,6 +22,8 @@ import com.google.common.base.Strings; import com.google.common.net.HostAndPort; + +import diskCacheV111.vehicles.MoverInfoMessage; import diskCacheV111.vehicles.ProtocolInfo; import java.io.IOException; import java.net.DatagramPacket; @@ -34,12 +36,14 @@ import java.time.Instant; import java.util.function.Function; import java.util.function.Predicate; +import java.util.Map; +import java.util.OptionalInt; +import java.util.HashMap; import javax.annotation.Nonnull; import javax.security.auth.Subject; import org.dcache.auth.Subjects; import org.dcache.net.FlowMarker.FlowMarkerBuilder; import org.dcache.util.IPMatcher; -import org.json.JSONObject; import org.slf4j.Logger; import org.slf4j.LoggerFactory; @@ -61,6 +65,9 @@ public class TransferLifeCycle { private Predicate localSubnet = a -> false; private boolean enabled; + private boolean storageStatisticsEnabled; + + private Map voToExpId = new HashMap<>(); /** * Mark transfer start. @@ -71,7 +78,7 @@ public class TransferLifeCycle { */ public void onStart(InetSocketAddress src, InetSocketAddress dst, ProtocolInfo protocolInfo, Subject subject) { - + if (!enabled) { return; } @@ -84,9 +91,14 @@ public void onStart(InetSocketAddress src, InetSocketAddress dst, ProtocolInfo p return; } + var optionalExpId = getExperimentId(protocolInfo, subject); + if (optionalExpId.isEmpty()) { + return; + } + var data = new FlowMarkerBuilder() .withStartedAt(Instant.now()) - .withExperimentId(getExperimentId(subject)) + .withExperimentId(optionalExpId.getAsInt()) .withActivityId(getActivity(protocolInfo)) .wittApplication(getApplication(protocolInfo)) .withProtocol("tcp") @@ -105,9 +117,11 @@ public void onStart(InetSocketAddress src, InetSocketAddress dst, ProtocolInfo p * @param protocolInfo access protocol information * @param subject associated with the transfer */ - public void onEnd(InetSocketAddress src, InetSocketAddress dst, ProtocolInfo protocolInfo, - Subject subject) { + public void onEnd(InetSocketAddress src, InetSocketAddress dst, MoverInfoMessage mover) { + ProtocolInfo protocolInfo = mover.getProtocolInfo(); + Subject subject = mover.getSubject(); + if (!enabled) { return; } @@ -120,19 +134,28 @@ public void onEnd(InetSocketAddress src, InetSocketAddress dst, ProtocolInfo pro return; } + var optionalExpId = getExperimentId(protocolInfo, subject); + if (optionalExpId.isEmpty()) { + return; + } + var data = new FlowMarkerBuilder() .withStartedAt(Instant.now()) .withFinishedAt(Instant.now()) - .withExperimentId(getExperimentId(subject)) + .withExperimentId(optionalExpId.getAsInt()) .withActivityId(getActivity(protocolInfo)) .wittApplication(getApplication(protocolInfo)) + .withUsage(mover.getBytesRead(), mover.getBytesWritten()) .withProtocol("tcp") .withAFI(toAFI(dst)) .withDestination(dst) - .withSource(src) - .build("end"); + .withSource(src); + if (storageStatisticsEnabled) { + data.withStats(mover); + } + var firefly = data.build("end"); - send(toFireflyDestination.apply(src), data); + send(toFireflyDestination.apply(src), firefly); } public void setFireflyDestination(String addr) { @@ -155,6 +178,33 @@ public void setEnabled(boolean isEnabled) { enabled = isEnabled; } + public void setStorageStatisticsEnabled(boolean isEnabled) { + storageStatisticsEnabled = isEnabled; + } + + /** + * Configures VO (Virtual Organization) to Experiment ID mapping. + * + * @param voMap A comma-separated string of VO mapping entries in the format + * "voName:expId". + */ + public void setVoMapping(String voMap) { + voToExpId.clear(); + + for (String entry : voMap.split(",")) { + String[] parts = entry.split(":"); + if (parts.length == 2) { + try { + voToExpId.put(parts[0].trim().toLowerCase(), Integer.parseInt(parts[1].trim())); + } catch (NumberFormatException e) { + LOGGER.warn("Invalid VO mapping entry: {}", entry); + } + } else { + LOGGER.warn("Invalid VO mapping entry: {}", entry); + } + } + } + /** * Send flow marker. * @@ -162,10 +212,10 @@ public void setEnabled(boolean isEnabled) { * @param payload the marker * @throws IllegalStateException if flow marker ist not build. */ - private void send(InetSocketAddress dst, @Nonnull JSONObject payload) + private void send(InetSocketAddress dst, @Nonnull String payload) throws IllegalStateException { - byte[] data = payload.toString().getBytes(StandardCharsets.UTF_8); + byte[] data = payload.getBytes(StandardCharsets.UTF_8); DatagramPacket p = new DatagramPacket(data, data.length); try (DatagramSocket socket = new DatagramSocket()) { socket.connect(dst); @@ -191,21 +241,38 @@ private String getApplication(ProtocolInfo protocolInfo) { return protocolInfo.getProtocol().toLowerCase(); } - private int getExperimentId(Subject subject) { + /** + * Determine experiment ID, initially from the ProtocolInfo (xroot/http), + * if that fails then fallback to the Subject's primary FQAN. + * + * @param protocolInfo the ProtocolInfo object containing transfer-related metadata + * @param subject the Subject representing the user or entity associated with the transfer + * @return the experiment ID, or -1 if it cannot be determined + */ + private OptionalInt getExperimentId(ProtocolInfo protocolInfo, Subject subject) { + if (protocolInfo.getTransferTag() != null && !protocolInfo.getTransferTag().isEmpty()) { + try { + int transferTag = Integer.parseInt(protocolInfo.getTransferTag()); + if (transferTag <= 64 || transferTag >= 65536) { + LOGGER.warn("Invalid integer range for transfer tag: {}", protocolInfo.getTransferTag()); + return OptionalInt.empty(); + } + // scitag = exp_id << 6 | act_id + return OptionalInt.of(transferTag >> 6); + } catch (NumberFormatException e) { + LOGGER.warn("Invalid transfer tag: {}", protocolInfo.getTransferTag()); + return OptionalInt.empty(); + } + } var vo = Subjects.getPrimaryFqan(subject); if (vo == null) { - return 0; + return OptionalInt.empty(); } - switch (vo.getGroup().toLowerCase()) { - case "atlas": - return 16; - case "cms": - return 23; - default: - return 0; - } + return voToExpId.containsKey(vo.getGroup().toLowerCase()) + ? OptionalInt.of(voToExpId.get(vo.getGroup().toLowerCase())) + : OptionalInt.empty(); } private boolean isLocalTransfer(InetSocketAddress dst) { @@ -214,8 +281,13 @@ private boolean isLocalTransfer(InetSocketAddress dst) { } private int getActivity(ProtocolInfo protocolInfo) { - // REVISIT: the activity should come from protocol info - return 14; // production + if (!protocolInfo.getTransferTag().isEmpty()) { + // scitag = exp_id << 6 | act_id + return Integer.parseInt(protocolInfo.getTransferTag()) & 0x3F; + } else { + // default activity id = 1 + return 1; + } } private String toAFI(InetSocketAddress dst) { diff --git a/modules/dcache/src/main/java/org/dcache/pool/repository/v5/ReplicaRepository.java b/modules/dcache/src/main/java/org/dcache/pool/repository/v5/ReplicaRepository.java index 12aa4f1524e..a00c41f02bc 100644 --- a/modules/dcache/src/main/java/org/dcache/pool/repository/v5/ReplicaRepository.java +++ b/modules/dcache/src/main/java/org/dcache/pool/repository/v5/ReplicaRepository.java @@ -838,8 +838,16 @@ public SpaceRecord getSpaceRecord() { SpaceRecord space = _account.getSpaceRecord(); long lru = (System.currentTimeMillis() - _sweeper.getLru()) / 1000L; long gap = _gap.orElse(Math.min(space.getTotalSpace() / 4, DEFAULT_GAP)); + + // REVISIT: This workaround addresses a filesystem behavior where the effective free space + // may be smaller than the total - used. + // To ensure correctness, we take the minimum of the disk reported and + // the dCache accounting expected free spaces. + long storeFreeSpace = _store.getFreeSpace(); + long effectiveFreeSpace = Math.min(storeFreeSpace, space.getFreeSpace()); + return new SpaceRecord(space.getTotalSpace(), - space.getFreeSpace(), + effectiveFreeSpace, space.getPreciousSpace(), space.getRemovableSpace(), lru, diff --git a/modules/dcache/src/main/java/org/dcache/util/Transfer.java b/modules/dcache/src/main/java/org/dcache/util/Transfer.java index 5e8f85de462..bc477bb47e3 100644 --- a/modules/dcache/src/main/java/org/dcache/util/Transfer.java +++ b/modules/dcache/src/main/java/org/dcache/util/Transfer.java @@ -28,6 +28,7 @@ import com.google.common.util.concurrent.MoreExecutors; import com.google.common.util.concurrent.ThreadFactoryBuilder; import diskCacheV111.poolManager.RequestContainerV5.RequestState; +import diskCacheV111.util.AccessLatency; import diskCacheV111.util.CacheException; import diskCacheV111.util.CheckStagePermission; import diskCacheV111.util.FileExistsCacheException; @@ -750,7 +751,7 @@ public ListenableFuture readNameSpaceEntryAsync(boolean allowWrite) { } private ListenableFuture readNameSpaceEntryAsync(boolean allowWrite, long timeout) { - Set attr = EnumSet.of(PNFSID, TYPE, STORAGEINFO, SIZE, CREATION_TIME); + Set attr = EnumSet.of(PNFSID, TYPE, STORAGEINFO, SIZE, CREATION_TIME, QOS_STATE); attr.addAll(_additionalAttributes); attr.addAll(PoolMgrSelectReadPoolMsg.getRequiredAttributes()); Set mask; @@ -925,13 +926,22 @@ protected synchronized void setReadPoolSelectionContext( */ public ListenableFuture selectPoolAsync(long timeout) { + FileAttributes fileAttributes = getFileAttributes(); + + // if this is a read, and the file has QoS policy nearline, + // then read is forbidden independent of is there cached copy or not. + if (!isWrite() && fileAttributes.isDefined(QOS_STATE) && + fileAttributes.getAccessLatency().equals(AccessLatency.NEARLINE)) { + return immediateFailedFuture( + new PermissionDeniedCacheException( + "Read is forbidden for file with QoS policy nearline")); + } + if (getPool() != null) { // we have a valid preselected pool. Let use it as long as clearPoolSelection is not called. return immediateFuture(null); } - FileAttributes fileAttributes = getFileAttributes(); - ProtocolInfo protocolInfo = getProtocolInfoForPoolManager(); ListenableFuture reply; if (isWrite()) { diff --git a/modules/dcache/src/main/java/org/dcache/util/jetty/ConnectorFactoryBean.java b/modules/dcache/src/main/java/org/dcache/util/jetty/ConnectorFactoryBean.java index 69b9fce49ee..72470789b90 100644 --- a/modules/dcache/src/main/java/org/dcache/util/jetty/ConnectorFactoryBean.java +++ b/modules/dcache/src/main/java/org/dcache/util/jetty/ConnectorFactoryBean.java @@ -308,6 +308,10 @@ public ServerConnector getObject() throws Exception { checkState(protocol == PLAIN || certificateAuthorityPath != null); HttpConnectionFactory httpConnectionFactory = new HttpConnectionFactory(); + /* + * disable reporting of Jetty version + */ + httpConnectionFactory.getHttpConfiguration().setSendServerVersion(false); switch (protocol) { case PLAIN: @@ -362,4 +366,4 @@ public boolean isSingleton() { public enum Protocol { PLAIN, TLS, GSI } -} \ No newline at end of file +} diff --git a/modules/dcache/src/main/resources/org/dcache/pool/classic/pool.xml b/modules/dcache/src/main/resources/org/dcache/pool/classic/pool.xml index 379490850b3..b36db8c04bb 100644 --- a/modules/dcache/src/main/resources/org/dcache/pool/classic/pool.xml +++ b/modules/dcache/src/main/resources/org/dcache/pool/classic/pool.xml @@ -350,6 +350,8 @@ + + allRepliesSentWith(CellEndpoint endpointUsed) { var envelopeArg = ArgumentCaptor.forClass(CellMessage.class); verify(endpointUsed, Mockito.atLeastOnce()).sendMessage(envelopeArg.capture()); diff --git a/modules/dcache/src/test/java/org/dcache/net/FlowMarkerTest.java b/modules/dcache/src/test/java/org/dcache/net/FlowMarkerTest.java index 1d4ba81d8dd..b538d274371 100644 --- a/modules/dcache/src/test/java/org/dcache/net/FlowMarkerTest.java +++ b/modules/dcache/src/test/java/org/dcache/net/FlowMarkerTest.java @@ -1,5 +1,7 @@ package org.dcache.net; +import static org.junit.Assert.assertTrue; + import java.io.IOException; import java.io.InputStream; import java.net.InetAddress; @@ -66,7 +68,7 @@ public void shouldAcceptFullyBuildStartMarker() { .withSource(new InetSocketAddress(InetAddress.getLoopbackAddress(), 5678)) .build("start"); - assertJson(data); + assertFirefly(data); } @Test @@ -83,7 +85,7 @@ public void shouldAcceptFullyBuildEndMarker() { .withSource(new InetSocketAddress(InetAddress.getLoopbackAddress(), 5678)) .build("end"); - assertJson(data); + assertFirefly(data); } @Test @@ -99,7 +101,7 @@ public void shouldAcceptFullyBuildOngoingMarker() { .withSource(new InetSocketAddress(InetAddress.getLoopbackAddress(), 5678)) .build("ongoing"); - assertJson(data); + assertFirefly(data); } @Test(expected = AssertionError.class) @@ -107,11 +109,14 @@ public void shouldFailWithEmptyMarker() { var data = new FlowMarkerBuilder() .build("start"); - assertJson(data); + assertFirefly(data); } - private void assertJson(JSONObject json) { + private void assertFirefly(String firefly) { + assertTrue(firefly.contains("dCache - firefly-json -")); + assertTrue(firefly.contains("<134>1")); + var json = new JSONObject(firefly.substring(firefly.indexOf("{"))); try { SCHEMA.validate(json); } catch (ValidationException e) { diff --git a/modules/ftp-client/pom.xml b/modules/ftp-client/pom.xml index 223038e120b..5f044753dc1 100644 --- a/modules/ftp-client/pom.xml +++ b/modules/ftp-client/pom.xml @@ -4,7 +4,7 @@ org.dcache dcache-parent - 10.2.0-SNAPSHOT + 10.2.12 ../../pom.xml diff --git a/modules/gplazma2-alise/pom.xml b/modules/gplazma2-alise/pom.xml index 1b7eb2a8f2d..4346387aebc 100644 --- a/modules/gplazma2-alise/pom.xml +++ b/modules/gplazma2-alise/pom.xml @@ -5,7 +5,7 @@ org.dcache dcache-parent - 10.2.0-SNAPSHOT + 10.2.12 ../../pom.xml diff --git a/modules/gplazma2-banfile/pom.xml b/modules/gplazma2-banfile/pom.xml index 996a035ec49..43e18c069ab 100644 --- a/modules/gplazma2-banfile/pom.xml +++ b/modules/gplazma2-banfile/pom.xml @@ -4,7 +4,7 @@ org.dcache dcache-parent - 10.2.0-SNAPSHOT + 10.2.12 ../../pom.xml diff --git a/modules/gplazma2-fermi/pom.xml b/modules/gplazma2-fermi/pom.xml index eee421a9fac..c3a3bddc5e5 100644 --- a/modules/gplazma2-fermi/pom.xml +++ b/modules/gplazma2-fermi/pom.xml @@ -5,7 +5,7 @@ dcache-parent org.dcache - 10.2.0-SNAPSHOT + 10.2.12 ../../pom.xml @@ -18,7 +18,7 @@ org.dcache gplazma2 - 10.2.0-SNAPSHOT + 10.2.12 com.google.code.gson diff --git a/modules/gplazma2-grid/pom.xml b/modules/gplazma2-grid/pom.xml index 7823e8bdaf9..2903a87b283 100644 --- a/modules/gplazma2-grid/pom.xml +++ b/modules/gplazma2-grid/pom.xml @@ -4,7 +4,7 @@ org.dcache dcache-parent - 10.2.0-SNAPSHOT + 10.2.12 ../../pom.xml diff --git a/modules/gplazma2-htpasswd/pom.xml b/modules/gplazma2-htpasswd/pom.xml index e2291e2b7e3..2b4f58011c9 100644 --- a/modules/gplazma2-htpasswd/pom.xml +++ b/modules/gplazma2-htpasswd/pom.xml @@ -4,7 +4,7 @@ org.dcache dcache-parent - 10.2.0-SNAPSHOT + 10.2.12 ../../pom.xml diff --git a/modules/gplazma2-jaas/pom.xml b/modules/gplazma2-jaas/pom.xml index 4c346a8e3a2..45fbe1be276 100644 --- a/modules/gplazma2-jaas/pom.xml +++ b/modules/gplazma2-jaas/pom.xml @@ -4,7 +4,7 @@ org.dcache dcache-parent - 10.2.0-SNAPSHOT + 10.2.12 ../../pom.xml diff --git a/modules/gplazma2-kpwd/pom.xml b/modules/gplazma2-kpwd/pom.xml index 2b818848801..0dd640dc842 100644 --- a/modules/gplazma2-kpwd/pom.xml +++ b/modules/gplazma2-kpwd/pom.xml @@ -4,7 +4,7 @@ org.dcache dcache-parent - 10.2.0-SNAPSHOT + 10.2.12 ../../pom.xml diff --git a/modules/gplazma2-krb5/pom.xml b/modules/gplazma2-krb5/pom.xml index 242d92ba2b5..16d180f8c9e 100644 --- a/modules/gplazma2-krb5/pom.xml +++ b/modules/gplazma2-krb5/pom.xml @@ -4,7 +4,7 @@ org.dcache dcache-parent - 10.2.0-SNAPSHOT + 10.2.12 ../../pom.xml diff --git a/modules/gplazma2-ldap/pom.xml b/modules/gplazma2-ldap/pom.xml index be93b6931ba..c54c155e306 100644 --- a/modules/gplazma2-ldap/pom.xml +++ b/modules/gplazma2-ldap/pom.xml @@ -5,7 +5,7 @@ org.dcache dcache-parent - 10.2.0-SNAPSHOT + 10.2.12 ../../pom.xml diff --git a/modules/gplazma2-multimap/pom.xml b/modules/gplazma2-multimap/pom.xml index 40a3677f519..94ad223b313 100644 --- a/modules/gplazma2-multimap/pom.xml +++ b/modules/gplazma2-multimap/pom.xml @@ -5,7 +5,7 @@ org.dcache dcache-parent - 10.2.0-SNAPSHOT + 10.2.12 ../../pom.xml diff --git a/modules/gplazma2-nis/pom.xml b/modules/gplazma2-nis/pom.xml index f5eed4793a7..cfe412000aa 100644 --- a/modules/gplazma2-nis/pom.xml +++ b/modules/gplazma2-nis/pom.xml @@ -4,7 +4,7 @@ org.dcache dcache-parent - 10.2.0-SNAPSHOT + 10.2.12 ../../pom.xml diff --git a/modules/gplazma2-nsswitch/pom.xml b/modules/gplazma2-nsswitch/pom.xml index 8d7baa3b10f..715a87a5720 100644 --- a/modules/gplazma2-nsswitch/pom.xml +++ b/modules/gplazma2-nsswitch/pom.xml @@ -4,7 +4,7 @@ org.dcache dcache-parent - 10.2.0-SNAPSHOT + 10.2.12 ../../pom.xml diff --git a/modules/gplazma2-oidc-te/pom.xml b/modules/gplazma2-oidc-te/pom.xml index 5094cc23976..3b95d14ab8a 100644 --- a/modules/gplazma2-oidc-te/pom.xml +++ b/modules/gplazma2-oidc-te/pom.xml @@ -5,7 +5,7 @@ org.dcache dcache-parent - 10.2.0-SNAPSHOT + 10.2.12 ../../pom.xml diff --git a/modules/gplazma2-oidc/pom.xml b/modules/gplazma2-oidc/pom.xml index 77764189f29..b5c62fa9582 100644 --- a/modules/gplazma2-oidc/pom.xml +++ b/modules/gplazma2-oidc/pom.xml @@ -5,7 +5,7 @@ org.dcache dcache-parent - 10.2.0-SNAPSHOT + 10.2.12 ../../pom.xml diff --git a/modules/gplazma2-omnisession/pom.xml b/modules/gplazma2-omnisession/pom.xml index 7fd512a857c..4314ced72bc 100644 --- a/modules/gplazma2-omnisession/pom.xml +++ b/modules/gplazma2-omnisession/pom.xml @@ -5,7 +5,7 @@ org.dcache dcache-parent - 10.2.0-SNAPSHOT + 10.2.12 ../../pom.xml diff --git a/modules/gplazma2-pyscript/pom.xml b/modules/gplazma2-pyscript/pom.xml index 297a8a00455..d9b88bcb0e2 100644 --- a/modules/gplazma2-pyscript/pom.xml +++ b/modules/gplazma2-pyscript/pom.xml @@ -4,7 +4,7 @@ org.dcache dcache-parent - 10.2.0-SNAPSHOT + 10.2.12 ../../pom.xml diff --git a/modules/gplazma2-roles/pom.xml b/modules/gplazma2-roles/pom.xml index 247bfe7c28c..2aecf23ae55 100644 --- a/modules/gplazma2-roles/pom.xml +++ b/modules/gplazma2-roles/pom.xml @@ -4,7 +4,7 @@ org.dcache dcache-parent - 10.2.0-SNAPSHOT + 10.2.12 ../../pom.xml diff --git a/modules/gplazma2-scitoken/pom.xml b/modules/gplazma2-scitoken/pom.xml index 8feea7ff1ac..df6670ecff4 100644 --- a/modules/gplazma2-scitoken/pom.xml +++ b/modules/gplazma2-scitoken/pom.xml @@ -5,7 +5,7 @@ org.dcache dcache-parent - 10.2.0-SNAPSHOT + 10.2.12 ../../pom.xml diff --git a/modules/gplazma2-voms/pom.xml b/modules/gplazma2-voms/pom.xml index d746661ec2a..3b70261c6b6 100644 --- a/modules/gplazma2-voms/pom.xml +++ b/modules/gplazma2-voms/pom.xml @@ -4,7 +4,7 @@ org.dcache dcache-parent - 10.2.0-SNAPSHOT + 10.2.12 ../../pom.xml diff --git a/modules/gplazma2/pom.xml b/modules/gplazma2/pom.xml index 5075e6283a8..6d6718f968b 100644 --- a/modules/gplazma2/pom.xml +++ b/modules/gplazma2/pom.xml @@ -4,7 +4,7 @@ org.dcache dcache-parent - 10.2.0-SNAPSHOT + 10.2.12 ../../pom.xml diff --git a/modules/logback-console-config/pom.xml b/modules/logback-console-config/pom.xml index 1ae86bf85b2..c1a215ef5dc 100644 --- a/modules/logback-console-config/pom.xml +++ b/modules/logback-console-config/pom.xml @@ -4,7 +4,7 @@ org.dcache dcache-parent - 10.2.0-SNAPSHOT + 10.2.12 ../../pom.xml diff --git a/modules/logback-test-config/pom.xml b/modules/logback-test-config/pom.xml index 52a164f280e..a67bd3d5bc3 100644 --- a/modules/logback-test-config/pom.xml +++ b/modules/logback-test-config/pom.xml @@ -3,7 +3,7 @@ org.dcache logback-test-config - 10.2.0-SNAPSHOT + 10.2.12 jar Logback config for building and testing diff --git a/modules/missingfiles-semsg/pom.xml b/modules/missingfiles-semsg/pom.xml index 1a186a24f71..a80c452b58f 100644 --- a/modules/missingfiles-semsg/pom.xml +++ b/modules/missingfiles-semsg/pom.xml @@ -5,7 +5,7 @@ org.dcache dcache-parent - 10.2.0-SNAPSHOT + 10.2.12 ../../pom.xml diff --git a/modules/srm-client/pom.xml b/modules/srm-client/pom.xml index 02b34958d96..6e470558705 100644 --- a/modules/srm-client/pom.xml +++ b/modules/srm-client/pom.xml @@ -4,7 +4,7 @@ org.dcache dcache-parent - 10.2.0-SNAPSHOT + 10.2.12 ../../pom.xml diff --git a/modules/srm-common/pom.xml b/modules/srm-common/pom.xml index e5029aad751..d766d912a72 100644 --- a/modules/srm-common/pom.xml +++ b/modules/srm-common/pom.xml @@ -4,7 +4,7 @@ org.dcache dcache-parent - 10.2.0-SNAPSHOT + 10.2.12 ../../pom.xml diff --git a/modules/srm-server/pom.xml b/modules/srm-server/pom.xml index 92b2e4ad021..614ef575c18 100644 --- a/modules/srm-server/pom.xml +++ b/modules/srm-server/pom.xml @@ -4,7 +4,7 @@ org.dcache dcache-parent - 10.2.0-SNAPSHOT + 10.2.12 ../../pom.xml diff --git a/packages/fhs/pom.xml b/packages/fhs/pom.xml index 9fc4a2733e0..e178a2b1dc5 100644 --- a/packages/fhs/pom.xml +++ b/packages/fhs/pom.xml @@ -4,7 +4,7 @@ org.dcache packaging - 10.2.0-SNAPSHOT + 10.2.12 dcache-fhs @@ -26,7 +26,7 @@ 1 - + scitags diff --git a/packages/pom.xml b/packages/pom.xml index b856d0c97d3..5fbef727439 100644 --- a/packages/pom.xml +++ b/packages/pom.xml @@ -4,7 +4,7 @@ org.dcache dcache-parent - 10.2.0-SNAPSHOT + 10.2.12 packaging @@ -198,7 +198,7 @@ org.dcache gplazma2-pyscript - 10.2.0-SNAPSHOT + 10.2.12 org.dcache diff --git a/packages/system-test/pom.xml b/packages/system-test/pom.xml index 108e4e3d5d1..b3024740678 100644 --- a/packages/system-test/pom.xml +++ b/packages/system-test/pom.xml @@ -4,7 +4,7 @@ org.dcache packaging - 10.2.0-SNAPSHOT + 10.2.12 system-test diff --git a/packages/tar/pom.xml b/packages/tar/pom.xml index acdb8be9ba1..0288f17e9e3 100644 --- a/packages/tar/pom.xml +++ b/packages/tar/pom.xml @@ -4,7 +4,7 @@ org.dcache packaging - 10.2.0-SNAPSHOT + 10.2.12 dcache-tar diff --git a/plugins/hsqldb/pom.xml b/plugins/hsqldb/pom.xml index 76e6e02a016..554a6bc2f8b 100644 --- a/plugins/hsqldb/pom.xml +++ b/plugins/hsqldb/pom.xml @@ -4,7 +4,7 @@ org.dcache plugins - 10.2.0-SNAPSHOT + 10.2.12 dcache-plugin-hsqldb diff --git a/plugins/pom.xml b/plugins/pom.xml index 3eda2e1bce1..5921b82157c 100644 --- a/plugins/pom.xml +++ b/plugins/pom.xml @@ -4,7 +4,7 @@ org.dcache dcache-parent - 10.2.0-SNAPSHOT + 10.2.12 plugins diff --git a/pom.xml b/pom.xml index a01f1a21e82..f58d0cf0a52 100644 --- a/pom.xml +++ b/pom.xml @@ -1,7 +1,7 @@ 4.0.0 org.dcache - 10.2.0-SNAPSHOT + 10.2.12 dcache-parent pom @@ -67,7 +67,7 @@ 1.9.21.2 6.6.0 2.12.0 - 9.4.52.v20230823 + 9.4.57.v20241219 4.6.0 2.41 3.0.0 @@ -107,7 +107,7 @@ https://github.com/dCache/dcache scm:git:https://github.com/dCache/dcache.git scm:git:ssh://git@github.com/dCache/dcache.git - 10.2 + 10.2.12 diff --git a/skel/share/defaults/dcache.properties b/skel/share/defaults/dcache.properties index 1d144ad72e9..a27fea6f589 100644 --- a/skel/share/defaults/dcache.properties +++ b/skel/share/defaults/dcache.properties @@ -198,7 +198,6 @@ dcache.log.qos.max-history=30 -XX:HeapDumpPath=${dcache.java.oom.file} \ -XX:+ExitOnOutOfMemoryError \ -XX:+StartAttachListener \ - -XX:+UseCompressedOops \ -javaagent:${dcache.paths.classes}/aspectjweaver-1.9.21.2.jar \ -Djava.net.preferIPv6Addresses=system \ --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.text=ALL-UNNAMED --add-opens=java.sql/java.sql=ALL-UNNAMED --add-opens=java.base/java.math=ALL-UNNAMED --add-opens java.base/sun.nio.fs=ALL-UNNAMED \ diff --git a/skel/share/defaults/pool.properties b/skel/share/defaults/pool.properties index 0f8d4d5bdc0..6eddb436e20 100644 --- a/skel/share/defaults/pool.properties +++ b/skel/share/defaults/pool.properties @@ -940,4 +940,17 @@ pool.firefly.destination= # # example: a.b.c.d/16, aa:bb:cc:dd:ee:ff/64, v.x.w.z # -pool.firefly.excludes= \ No newline at end of file +pool.firefly.excludes= + +# +# A comma separated list of virtual organisations and their mappings to Scitags experiment IDs +# Experiment IDs must be based on https://scitags.docs.cern.ch/api.json +# +# example: atlas:2, cms:3,etc. +# +pool.firefly.vo-mapping=atlas:2, cms:3, lhcb:4, alice:5, belleii:6, ska:7, dune:8, lsst:9, ilc:10, auger:11, juno:12, nova:13, xenon:14 + +# +# Enable sending storage statistics as part of the firefly packets +# +pool.firefly.storage-statistics=false diff --git a/skel/share/httpd/static/images/dCache-logo.svg b/skel/share/httpd/static/images/dCache-logo.svg new file mode 100644 index 00000000000..868008b704d --- /dev/null +++ b/skel/share/httpd/static/images/dCache-logo.svg @@ -0,0 +1,131 @@ + + + + + + + + + + + + + + + + + image/svg+xml + + + + + + + + + + + diff --git a/skel/share/httpd/static/images/eagleredtrans.gif b/skel/share/httpd/static/images/eagleredtrans.gif deleted file mode 100644 index ec548cbc854..00000000000 Binary files a/skel/share/httpd/static/images/eagleredtrans.gif and /dev/null differ diff --git a/skel/share/httpd/static/index.html b/skel/share/httpd/static/index.html index 683add8df81..9d4a48180f2 100644 --- a/skel/share/httpd/static/index.html +++ b/skel/share/httpd/static/index.html @@ -4,71 +4,51 @@ dCache service - + - +

-

dCache service

+

dCache service

+


-
Pool Group : "); pw.print(groupName); pw.println("
+
-
+

Quick Finder

- +
- - - - - - - - -
+

Cell Services

 +

Pool Usage

 +

Tape Transfer Queue

 +

Detailed Tape Transfer Queue
+

Pool Transfer Queues

 +

Action Log

 + + @@ -79,10 +59,10 @@

History
+

Status

- +
-

Cell Services

@@ -112,10 +92,10 @@

Pool Groups

 +

Various Queues

- +

Tape Transfer Queue

@@ -146,10 +126,10 @@

Active Transfers  +

Statistics Module not installed

- +
-

Statistics

@@ -163,10 +143,10 @@

Statistics

 +

Configuration/Setup

- +
-

Pool Selection Setup

@@ -181,7 +161,7 @@

Pool Selection Setup

+

dCache, the Book

diff --git a/skel/share/httpd/static/styles/billing.css b/skel/share/httpd/static/styles/billing.css index b499e00a23d..2b7bd1ca710 100644 --- a/skel/share/httpd/static/styles/billing.css +++ b/skel/share/httpd/static/styles/billing.css @@ -1,7 +1,3 @@ -#bird_small -{ - float: left; -} #bird_large { @@ -11,7 +7,7 @@ h1, h2 { text-align: center; - color: blue; + color: #3C3633; } table @@ -23,13 +19,13 @@ table tr.total th, tr.total td { - background: #0000FF; - color: white; + background: #747264; + color: #EEEDEB; } a { - color: white; + color: #EEEDEB; } .action diff --git a/skel/share/httpd/static/styles/common.css b/skel/share/httpd/static/styles/common.css index a8d6506cb43..2d20ba0b29f 100644 --- a/skel/share/httpd/static/styles/common.css +++ b/skel/share/httpd/static/styles/common.css @@ -1,23 +1,62 @@ -body -{ +:root { + --v-page-background-color: #f1efed; + --v-page-text-color: #222831; + --v-table-border-color: rgba(64, 60, 56, 0.5); + --v-table-header-background-color: #403c38; + --v-tr-odd-background-color: #e9e9e9; + --v-tr-even-background-color: #f6f6f6; + --v-page-header-background-color: #403c38; + --v-page-header-text-color: #ffffff; + --v-page-header-separator-color: #e3513c; + + --v-index-container-background-color: #e9e9e9; + + --v-link-hover-color: #e3513c; + + --v-space-precious: #F28585; + --v-space-sticky: #FFA447; + --v-space-cached: #FFFC9B; + --v-space-free: #B7E5B4; + +} + +#index-page +{ background: url("/images/bg.svg"); + background-color: var(--v-page-background-color); + color: var(--v-page-text-color); + margin: 0px; } -img -{ + +body +{ + /*background: url("/images/bg.svg"); */ + background-color: var(--v-page-background-color); + color: var(--v-page-text-color); + margin: 0px; +} + +img +{ border: 0; } #bird_small { - color: red; + color: var(--v-page-header-text-color); + background-color: var(--v-page-header-background-color); + padding: 1rem; + border-bottom: 4px solid var(--v-page-header-separator-color); } #bird_large { text-align: center; + display: none; } + #main { margin-left: auto; @@ -30,29 +69,77 @@ img margin-top: 200px; } + +#index-page-table td +{ + text-align: left!important; + padding-left: 1rem; + padding-right: 1rem; +} + +#index-page-table tr +{ + height: 1rem; +} + +#index-container td +{ + background-color: var(--v-index-container-background-color); + a:link { color: var(--v-page-text-color); }; + a:focus { color: var(--v-link-hover-color); }; + a:visited { color: var(--v-page-text-color); }; + a:hover { color: var(--v-link-hover-color); }; +} + +#index-page-table h1 +{ + text-align: center!important; + padding-left: 1rem; + padding-right: 1rem; +} + +#index-page-table +{ + width: 90%; +} + +table, tr, th, td { + border-collapse: collapse; + border: 0px solid; + height: 3rem; +} + table -{ +{ + border-collapse: separate; border-spacing: 0; - border-collapse: collapse; + outline: 1px solid var(--v-table-border-color); + overflow: hidden; width: 100%; -} + color: var(--v-page-text-color); + /*margin: 0 0 40px 0;*/ +}; + td, th { - padding: 4px; - border: 1px solid gray; + padding: .5em 1em; + border: 0px solid gray; + margin-bottom: 10px; + padding: 0; + height: 6px; } th { text-align: center; - background: #115259; - color: white; + background: var(--v-table-header-background-color); + color: #ffffff; } th > a { - color: white; + color: #ffffff; text-decoration: none; } @@ -63,12 +150,66 @@ td tr { - background: #bebebe; + background: var(--v-tr-even-background-color); } tr.odd { - background: #efefef; + background: var(--v-tr-odd-background-color); +} + + +/* used to display pool/poolgroup usage bar and legend*/ + +span.layout_precious +{ + color: var(--v-space-precious); +} + +span.layout_sticky +{ + color: var(--v-space-sticky); +} + +span.layout_cached +{ + color: var(--v-space-cached); } +span.layout_free +{ + color: var(--v-space-free); +} + +div.layout_precious +{ + float: left; + background: var(--v-space-precious); + height: 10px; +} +div.layout_sticky +{ + float: left; + background: var(--v-space-sticky); + height: 10px; +} + +div.layout_cached +{ + float: left; + background: var(--v-space-cached); + height: 10px; +} + +div.layout_free +{ + float: left; + background: var(--v-space-free); + height: 10px; +} + +.errorcode, .errormessage +{ + color: red; +} diff --git a/skel/share/httpd/static/styles/poolinfo.css b/skel/share/httpd/static/styles/poolinfo.css index 833f47d6c27..2e73dc15cb1 100644 --- a/skel/share/httpd/static/styles/poolinfo.css +++ b/skel/share/httpd/static/styles/poolinfo.css @@ -1,8 +1,3 @@ -#bird_small -{ - float: left; -} - #bird_large { display: none; @@ -55,28 +50,6 @@ table.menu .active a:link color: red; } -/* Pool info */ - -span.layout_precious -{ - color: red; -} - -span.layout_sticky -{ - color: magenta; -} - -span.layout_cached -{ - color: green; -} - -span.layout_free -{ - color: yellow; -} - td.layout > div { margin-left: auto; @@ -85,58 +58,21 @@ td.layout > div height: 10px; } -div.layout_precious -{ - float: left; - background: red; - height: 10px; -} - -div.layout_sticky -{ - float: left; - background: #ff00ff; - height: 10px; -} - -div.layout_cached -{ - float: left; - background: green; - height: 10px; -} - -div.layout_free -{ - float: left; - background: yellow; - height: 10px; -} - -.errorcode, .errormessage -{ - color: red; -} - /* Queue info */ th { font-weight: normal; } -th.active, th.max -{ - background: #0099FF; -} - -th.queued +th.active, th.max, th.queued { - background: #00bbFF; + background: #E0CCBE; + color: #3C3633; } td.idle { - color: #008080; + color: #3C3633; } td.queued @@ -146,6 +82,6 @@ td.queued tr.total th, tr.total td { - background: #0000FF; + background: #747264; color: white; } diff --git a/skel/share/httpd/static/styles/queueInfo.css b/skel/share/httpd/static/styles/queueInfo.css index c9f073c44e1..b6528ebd666 100644 --- a/skel/share/httpd/static/styles/queueInfo.css +++ b/skel/share/httpd/static/styles/queueInfo.css @@ -3,19 +3,15 @@ th font-weight: normal; } -th.active, th.max +th.active, th.max, th.queued { - background: #0099FF; -} - -th.queued -{ - background: #00bbFF; + background: #FFA27F; + color: #3C3633; } td.idle { - color: #008080; + color: #3C3633; } td.queued @@ -25,6 +21,6 @@ td.queued tr.total th, tr.total td { - background: #0000FF; - color: white; + background: #FFE8C5; + color: #3C3633; } diff --git a/skel/share/httpd/static/styles/restoreHandler.css b/skel/share/httpd/static/styles/restoreHandler.css index b0f11b00734..732eca6cbd9 100644 --- a/skel/share/httpd/static/styles/restoreHandler.css +++ b/skel/share/httpd/static/styles/restoreHandler.css @@ -1,8 +1,3 @@ -#bird_small -{ - float: left; -} - #bird_large { display: none; @@ -16,7 +11,6 @@ h1, h2 { text-align: center; - color: blue; } h1 diff --git a/skel/share/httpd/static/styles/usageInfo.css b/skel/share/httpd/static/styles/usageInfo.css index 1cadfd5e674..c52936a9b42 100644 --- a/skel/share/httpd/static/styles/usageInfo.css +++ b/skel/share/httpd/static/styles/usageInfo.css @@ -1,22 +1,4 @@ -span.layout_precious -{ - color: red; -} - -span.layout_sticky -{ - color: magenta; -} - -span.layout_cached -{ - color: green; -} -span.layout_free -{ - color: yellow; -} td.layout > div { @@ -25,36 +7,3 @@ td.layout > div width: 300px; height: 10px; } - -div.layout_precious -{ - float: left; - background: red; - height: 10px; -} - -div.layout_sticky -{ - float: left; - background: #ff00ff; - height: 10px; -} - -div.layout_cached -{ - float: left; - background: green; - height: 10px; -} - -div.layout_free -{ - float: left; - background: yellow; - height: 10px; -} - -.errorcode, .errormessage -{ - color: red; -} diff --git a/skel/share/lib/billing.sh b/skel/share/lib/billing.sh index a619e87c609..5ae4c87bfa9 100644 --- a/skel/share/lib/billing.sh +++ b/skel/share/lib/billing.sh @@ -2,7 +2,7 @@ billing_indexer() # $* = arguments { CLASSPATH="$(printLimitedClassPath slf4j-api logback-classic logback-core logback-console-config \ - jul-to-slf4j commons-compress gson spring-core guava dcache-common common-cli cells dcache-core \ + jul-to-slf4j commons-compress commons-io gson spring-core guava dcache-common common-cli cells dcache-core \ curator-client)" \ quickJava \ "-Ddcache.home=${DCACHE_HOME}" \