Parallel long running irods jobs result in conflicting session refresh tokens. Thus, py4lexis is limited to 1 job that takes "longer" per user

I have several processes running in parallel, accessing IRODS via a single account. This works fine until a certain point in time, where the session times out >10h). After the timeout, it seems the refresh tokens are conflicting and disconnects irods session from Lexis sessions, producing and ExceptionOpenIDAuthUrl: https://aai.lexis.tech/auth/realms/LEXIS_AAI_v2/protocol

Conclusion / Evidence:

• Based on my tests below, where two jobs run producing ExceptionOpenIDAuthUrl, starting a third process (my script below), that basically does something like:

lexis_session = LexisSession(suppress_print=False, in_cli=True,offline_access=True,login_method="url")
print(iRODS(session=lexis_session).get_dataset_collection(access="public", project="openwebsearch", dataset_id=d["dataset_id"]))

Produces

2025-07-18 09:00:05,867 -- ERROR -- py4lexis.core.lexis_irods -- log_and_print_errors -- iRODS Exception: error checking session validity

• After stopping the other two processes, the above script runs normal immediately.
• My hypothesis is, that the two parallel running jobs do refresh the irods token regularly, but then produce a different session token every time.
• The problem is, that this limits us to one long standing job per user

Details

I have a long running code (>12h) that regularly access files via irods. Everything is working fine, except that in some point in time (around >10h), irods starts throwing an ExceptionOpenIDAuthUrl: https://aai.lexis.tech/auth/realms/LEXIS_AAI_v2/protocol exception

I traced it down to the following code part in the iRODS() class:

   irods_session: iRODSSession = iRODSSession(host=self.__session.irods_path if self._host is None else self._host,
                                                       port=int(self.__Clr.yhbrr(_irds_prt)) if self._port is None else self._port,
                                                       authentication_scheme=self.__Clr.yhbrr(_sc),
                                                       openid_provider=self.__Clr.yhbrr(_prvdr),
                                                       zone=self._default_zone,
                                                       access_token=self.__session.get_access_token(),
                                                       user=self.__session._tokens.username,
                                                       block_on_authURL=False)

            if not self.__session.suppress_print:
                print(f"The iRODS session was successfully initialised.")

In the debugger, a new irods_session gets created. I have a breakpoint at if not self.__session.supres.... Now when calling irods_session.server_version at this breakpoint, i get the above exception. This indicates, that the session has not been correctly created or there is a mismatch between irods and lexis.

Further test (note that i have overwritten the iRODS() class, so _iRODS__session is the iRODS.session, i.e. the LexisSession) :

• LexisSession is still valid: Datasets(session=self._iRODS__session).get_all_datasets() returns a list of datasets
• Creating a new iRODS object yields the same error: iRODS(session=self._iRODS__session)._irds.server_version
• Creating a new iRODS object with a new Lexis Session using the offline token yields the same error: `iRODS(session=LexisSession(suppress_print=False, in_cli=True,offline_access=True,login_method="offline", refresh_token="my current offline token"))._irds.server_version

I ran the following script (with my offline token):

from py4lexis.core.lexis_irods import iRODS
from py4lexis.core.session import LexisSession
from py4lexis.ddi.datasets import Datasets


lexis_session = LexisSession(suppress_print=False, in_cli=True,offline_access=True,login_method="offline",
                              refresh_token="my offline token is ")
ds = Datasets(session=lexis_session, suppress_print=False).get_all_datasets()
print(ds)
v= iRODS(session=lexis_session)._irds.server_version
print(v)

Output

Connected to pydev debugger (build 251.26094.141)
Welcome to the Py4Lexis!
You have been successfully logged in LEXIS session.
Retrieving data of the datasets...
Data of the datasets successfully retrieved (and converted)....
......some datasets come here from both, lrz and it4i zones.................
Validating token on iRODS...
Validate token on iRODS was successfull...
The iRODS session was successfully initialised.
Traceback (most recent call last):
  File "/home/mgrani/bin/miniforge3/envs/owilix-dev/lib/python3.11/site-packages/irods/session.py", line 167, in server_version
    return tuple(ast.literal_eval(reported_vsn))
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mgrani/bin/miniforge3/envs/owilix-dev/lib/python3.11/ast.py", line 64, in literal_eval
    node_or_string = parse(node_or_string.lstrip(" \t"), mode='eval')
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mgrani/bin/miniforge3/envs/owilix-dev/lib/python3.11/ast.py", line 50, in parse
    return compile(source, filename, mode, flags,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<unknown>", line 0
    
SyntaxError: invalid syntax

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/mgrani/bin/miniforge3/envs/owilix-dev/lib/python3.11/site-packages/irods/session.py", line 173, in __server_version
    conn = next(iter(self.pool.active))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
StopIteration

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/mgrani/bin/miniforge3/envs/owilix-dev/lib/python3.11/site-packages/irods/pool.py", line 61, in get_connection
    conn = self.idle.pop()
           ^^^^^^^^^^^^^^^
KeyError: 'pop from an empty set'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/mgrani/git/3.ProAndPhD/2209-OWSEU/owi-cli/tests/irods_connection_openidauthurl.py", line 10, in <module>
    v= iRODS(session=lexis_session)._irds.server_version
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mgrani/bin/miniforge3/envs/owilix-dev/lib/python3.11/site-packages/irods/session.py", line 169, in server_version
    return self.__server_version()
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mgrani/bin/miniforge3/envs/owilix-dev/lib/python3.11/site-packages/irods/session.py", line 176, in __server_version
    conn = self.pool.get_connection()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mgrani/bin/miniforge3/envs/owilix-dev/lib/python3.11/site-packages/irods/pool.py", line 15, in method_
    ret = method(self,*s,**kw)
          ^^^^^^^^^^^^^^^^^^^^
  File "/home/mgrani/bin/miniforge3/envs/owilix-dev/lib/python3.11/site-packages/irods/pool.py", line 77, in get_connection
    conn = Connection(self, self.account, block_on_authURL=self.block_on_authURL)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mgrani/bin/miniforge3/envs/owilix-dev/lib/python3.11/site-packages/irods/connection.py", line 94, in __init__
    self._login_openid()
  File "/home/mgrani/bin/miniforge3/envs/owilix-dev/lib/python3.11/site-packages/irods/connection.py", line 565, in _login_openid
    raise ExceptionOpenIDAuthUrl(first)
irods.connection.ExceptionOpenIDAuthUrl: error checking session validity

I then added a loop to check every dataset with the irods collection:

from py4lexis.core.lexis_irods import iRODS
from py4lexis.core.session import LexisSession
from py4lexis.ddi.datasets import Datasets


lexis_session = LexisSession(suppress_print=False, in_cli=True,offline_access=True,login_method="offline",
                              refresh_token="token ommitted")
ds = Datasets(session=lexis_session, suppress_print=False).get_all_datasets()
print(ds[0:2])
for d in ds:
    print(d["dataset_id"])
    print(iRODS(session=lexis_session).get_dataset_collection(access="public", project="openwebsearch", dataset_id=d["dataset_id"]))
v= iRODS(session=lexis_session)._irds.server_version
print(v)

but it tells me now (in the lexis log)

025-07-18 08:59:04,637 -- ERROR -- py4lexis.core.lexis_irods -- log_and_print_errors -- iRODS Exception: error checking session validity
2025-07-18 08:59:04,638 -- INFO -- py4lexis.core.lexis_irods -- __validate_irds -- iRODS -- VALIDATE TOKEN -- PROGRESS
2025-07-18 08:59:04,638 -- INFO -- py4lexis.core.session -- make_request -- Preparing 'GET' request with no payload.
2025-07-18 08:59:04,842 -- INFO -- py4lexis.core.lexis_irods -- __validate_irds -- iRODS -- VALIDATE TOKEN -- OK
2025-07-18 08:59:04,842 -- INFO -- py4lexis.core.lexis_irods -- __get_irds_session -- iRODS -- INITIALISED -- OK
2025-07-18 08:59:04,842 -- INFO -- py4lexis.core.lexis_irods -- get_dataset_collection -- Getting directory (collection): /IT4ILexisV2/public/proj862c5962623246664c1fda27b7afb108/ed9825ba-6317-11f0-ab83-528c047b29ff
2025-07-18 08:59:04,843 -- INFO -- py4lexis.core.lexis_irods -- __get_collection_with_log -- GET iRODS collection: /IT4ILexisV2/public/proj862c5962623246664c1fda27b7afb108/ed9825ba-6317-11f0-ab83-528c047b29ff

So the session is no longer valid for iRODS, but it is ok for Lexis. The same happens for the URL login method. Does not work.

lexis_session = LexisSession(suppress_print=False, in_cli=True,offline_access=True,login_method="url")

Now i killed the other two processes that have been running under the same account, and the same script works as intended