Using dynamic selectors and artificial intelligence to harden web data retrieval codes
Pages 1-24
https://doi.org/10.22054/jks.2026.89870.1753
farnaz taghizadeh kourayem, Mohammadreza Kabaranzad Ghadim, Seyed Abdollah Amin Mousavi
Abstract Introduction
The robustness of web scraping systems against structural changes in web pages remains a major challenge. Selenium-based solutions are highly vulnerable because they depend on fixed selectors. Existing robustness approaches often suffer from high computational costs or low reliability. To address this gap, this paper proposes an AI-based self-healing approach that dynamically analyzes element behavior and automatically selects the most suitable alternative selector, enabling continuous and reliable data extraction without manual intervention.
Literature Review
The studies by Kirinuki et al. (2019), Nass et al. (2023), and Coppola et al. (2025) propose alternative selectors to improve the reliability of web data extraction code using various cues. However, these methods exhibit relatively high error rates and are not suitable for pages with significant structural changes or elements with similar attributes. Although the use of large language models was proposed by Nass et al. (2024) to improve these methods and sometimes reduce errors, it raises privacy concerns and increases computational costs. In contrast, the present paper proposes a method that not only reduces errors but is also effective for pages with major structural changes while avoiding privacy and computational cost issues.
The works of Degaki et al. (2022), Khaliq et al. (2023), and Daneshvar and Wang (2024) propose using computer vision to predict elements on web pages. However, computer vision techniques show low accuracy in detection and prediction when the target information is limited, and they also involve high computational and time costs. In contrast, the present paper introduces a method that retrieves precise information while minimizing computational and time costs.
Taghizadeh et al. (2024) present a mechanism for time management and improving the accuracy of information retrieval from web platforms using Selenium. However, their method only identifies the location of errors and returns erroneous values as null, without providing a solution for code robustness. In contrast, the present paper builds on the method proposed by Taghizadeh et al. (2024) as a baseline for web data extraction and offers a solution for code robustness and automatic extraction of target values with the highest possible accuracy.
Saarathy et al. (2024) conducted a systematic review of existing studies on Selenium code robustness but did not propose a robustness model for web data extraction. In contrast, the present paper reviews existing studies and models and proposes a dedicated robustness model for Selenium code.
Healenium (2025) stores successful Selenium element information for future use but still requires manual intervention when errors occur. In contrast, the present paper automatically identifies alternative scripts using AI algorithms, reducing recovery time and human involvement.
Kluge and Stocco (2025) concluded that robust web scraping requires hybrid methods and automated validation, but they did not propose a new model. In contrast, the present paper introduces a model designed to overcome the limitations of existing approaches.
Methodology
This research presents an automated and robust framework for sustainable web data extraction (web scraping) that aims to overcome the inherent fragility of tools such as Selenium when facing structural changes in web pages. The proposed method is a hybrid system that integrates a relational database (SQL), Python and Selenium scripts, and artificial intelligence algorithms (particularly time-series models) to automatically identify and replace faulty selectors (XPath/CSS).
Data Infrastructure Design and Collection
Database: A SQL database was designed for structured storage, consisting of three main components:
Products table: stores extracted product information (name, price, discount, rating, image, label, and date).
HTML Tags table: records the complete tree structure (parent–child relationships) of HTML tags from the target website’s product pages (e.g., Digikala). This table includes the XPath (XML Path Language) of each tag, the extraction date, and a Boolean flag called “used”, which indicates whether a given XPath has previously been used successfully to extract a target element (e.g., a product name).
Tag Count View (virtual table): calculates the frequency of each unique XPath in the HTML tags table. This view enables the algorithm to prioritize XPaths that appear an expected number of times (e.g., 20 times on a 20-product page) as candidates for repeated elements (such as product names).
Extraction and Retrieval Engine
Primary Extraction Script (Selenium): A Selenium-based script was implemented to extract product information from target web pages using initially defined fixed selectors.
Structure Retrieval and Analysis Module: If the primary script encounters an error or returns empty results, this indicates a possible structural change in the page. In such cases, a backend script is triggered that:
Reads the full HTML structure of the page and processes it after removing static sections (e.g., headers and footers).
Uses regular-expression (Regex) algorithms to extract the full XPath of all tags and stores them in the HTML tags table.
Intelligent Decision-Making and Automatic Repair Layer
AI Models (Time Series): The core of the repair system is a predictive model that applies time-series algorithms (such as LSTM) to analyze the historical behavior of the “used” flag for each XPath. The model was trained on six months of historical data initially collected and labeled through manual supervision.
Iterative Repair Process: When an error occurs:
The model predicts and proposes replacement XPaths with a high probability of being marked as “used” in the future.
Candidate XPaths are inserted into the Selenium script in priority order, and the script is re-executed.
If execution succeeds, the corresponding flag is updated in the database. If it fails, unsuccessful candidates are discarded and the process continues with the next options.
If no solution is found, the system falls back to a full rule-based traversal of the tags table.
Validation and Output Quality Control
To ensure the accuracy of data extracted using alternative selectors, the system incorporates validation rules:
Quantitative validation: The number of extracted items for repeated elements (e.g., product names) must match the expected count (e.g., 20).
Qualitative validation (data patterns): Extracted data must conform to predefined patterns (e.g., product names as strings within a maximum length, discount percentages as numbers followed by a % sign, and labels from predefined categories).
Experimental Evaluation
The system was evaluated on the Digikala website over a two-year period with weekly intervals. Evaluation metrics included extraction success rate, requirement for manual intervention, execution time, and prediction model accuracy. Results indicate that the proposed system achieved near-complete stability, eliminated the need for manual correction, and reduced the worst-case average execution time to approximately 9 minutes for 60 products, demonstrating significantly higher efficiency compared to alternative methods (e.g., image-based approaches). The LSTM model achieved very high accuracy in predicting correct selectors in many cases.
Conclusion
This study presents an automated, AI-based approach for robust web scraping. By analyzing the HTML structure, systematically extracting XPaths, and employing intelligent decision-making algorithms (such as LSTM) to dynamically select alternative selectors, the inherent fragility of tools such as Selenium is mitigated. The results demonstrate that this method can significantly reduce the data extraction failure rate and eliminate the need for manual intervention. The proposed system, by learning from historical data, evolves into a semi-autonomous web crawler that ensures the sustainability of knowledge and data retrieval systems in organizational and research applications.
