The weather API extractor

The extractor is the riskiest part of any pipeline. Network failures, rate limits, partial responses, and intermittent timeouts all live here. The Lambda we ship has retries, structured logging, and content-type validation so the failure modes are visible instead of silent.

lambda/weather_api.py
python
class WeatherAPI:
  def __init__(self, api_token):
    self.api_token = api_token
    self.base_url = "https://api.weatherapi.com/v1"

  async def _make_request(self, endpoint, params=None):
    url = f"{self.base_url}{endpoint}"
    if params is None:
      params = {}
    params.update({"key": self.api_token})

    try:
      async with aiohttp.ClientSession() as session:
        async with session.get(url, params=params) as response:
          response.raise_for_status()
          if "application/json" in response.headers.get('Content-Type', ''):
            data = await response.json()
          else:
            text = await response.text()
            raise ValueError(f"The response is not in json format. Content: {text}")
          return data
    except aiohttp.ClientError as e:
      print(f"Error fetching data from Weather API: {e}")
      return None

The WeatherAPI client wraps three endpoints (current, forecast, history). Note the explicit content-type check: APIs sometimes return HTML on errors and JSON parsers fail confusingly.

aiohttp lets one Lambda invocation fan out to many locations concurrently. With sequential requests, every location takes its own round trip and the latency stacks. Async drops the wall-clock cost to a single round trip latency. That matters because Lambda billing is per millisecond.

Both, but at different scopes. The Lambda retries transient HTTP errors with exponential backoff (single attempt, in-process). The orchestrator (Airflow) retries the whole Lambda invocation with longer backoff for systemic failures. Each layer handles a different timescale.

Code playground: Add exponential backoff to a request

Loading practice…