← blog

seo & crawlers

6/20/2025•3 min read

seo & crawlers

seo = getting found on google. matters for organic traffic, credibility, and long-term roi.

how google works

three stages:

1. crawling (discovery)

googlebot finds pages by:

  • following links from known pages
  • reading sitemaps
  • previous crawl data

crawl rate depends on:

  • site speed
  • authority
  • internal linking
  • content freshness

2. indexing

google analyzes and stores content:

  • reads text, images, videos
  • determines topics
  • checks quality & uniqueness
  • processes structured data

3. ranking

algorithm sorts indexed pages by:

  • relevance
  • page speed
  • mobile-friendliness
  • user experience
  • backlinks
  • E-A-T signals

E-A-T (expertise, authoritativeness, trustworthiness)

critical for ranking, especially YMYL content (health, finance, safety).

expertise: show you know your stuff

  • deep, researched content
  • author bios with credentials
  • cite sources
  • use proper terminology

authoritativeness: be the go-to source

  • earn quality backlinks
  • get mentioned by industry sites
  • strong social presence
  • contribute to publications

trustworthiness: be reliable

  • use HTTPS
  • clear privacy policy
  • show contact info
  • include reviews/testimonials
  • keep info accurate

technical SEO basics

  • logical URL structure
  • clean internal linking
  • XML sitemaps
  • fast page speed
  • mobile-first design
  • core web vitals (LCP, FID, CLS)

SEO in next.js

meta tags

import Head from 'next/head';

function SEOHead({ title, description, canonicalUrl, ogImage }) {
  return (
    <Head>
      <title>{title}</title>
      <meta name="description" content={description} />

      {/* open graph */}
      <meta property="og:title" content={title} />
      <meta property="og:description" content={description} />
      <meta property="og:image" content={ogImage} />

      {/* twitter */}
      <meta name="twitter:card" content="summary_large_image" />

      {/* canonical */}
      <link rel="canonical" href={canonicalUrl} />
    </Head>
  );
}

structured data

export default function BlogPost({ post }) {
  const structuredData = {
    "@context": "https://schema.org",
    "@type": "BlogPosting",
    "headline": post.title,
    "author": {
      "@type": "Person",
      "name": post.author
    },
    "datePublished": post.date,
  };

  return (
    <>
      <Head>
        <script
          type="application/ld+json"
          dangerouslySetInnerHTML={{ __html: JSON.stringify(structuredData) }}
        />
      </Head>
      <article>{post.content}</article>
    </>
  );
}

sitemap

// pages/api/sitemap.xml.js
export default async function handler(req, res) {
  const posts = await getPosts();

  const sitemap = `<?xml version="1.0" encoding="UTF-8"?>
    <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
      <url>
        <loc>https://yoursite.com</loc>
        <lastmod>${new Date().toISOString()}</lastmod>
        <priority>1.0</priority>
      </url>
      ${posts.map(post => `
        <url>
          <loc>https://yoursite.com/blog/${post.slug}</loc>
          <lastmod>${post.date}</lastmod>
          <priority>0.8</priority>
        </url>
      `).join('')}
    </urlset>`;

  res.setHeader('Content-Type', 'text/xml');
  res.write(sitemap);
  res.end();
}

robots.txt

// pages/api/robots.txt.js
export default function handler(req, res) {
  res.setHeader('Content-Type', 'text/plain');
  res.write(`User-agent: *
Allow: /
Sitemap: https://yoursite.com/api/sitemap.xml`);
  res.end();
}

image optimization

import Image from 'next/image';

// automatic optimization
<Image
  src="/hero.jpg"
  alt="descriptive alt text"
  width={1200}
  height={630}
  priority // for above-fold images
/>

SEO in node.js/express

meta tags middleware

app.get('/blog/:slug', async (req, res) => {
  const post = await getPost(req.params.slug);

  res.send(`
    <!DOCTYPE html>
    <html>
      <head>
        <title>${post.title}</title>
        <meta name="description" content="${post.excerpt}" />
        <meta property="og:title" content="${post.title}" />
        <meta property="og:image" content="${post.image}" />
      </head>
      <body>${post.content}</body>
    </html>
  `);
});

compression

const compression = require('compression');
app.use(compression());

caching

app.get('/api/data', (req, res) => {
  res.set('Cache-Control', 'public, max-age=3600');
  res.json(data);
});

core web vitals optimization

LCP (largest contentful paint) - < 2.5s

  • optimize images
  • preload key resources
  • minimize CSS/JS
  • use CDN
// preload critical resources
<Head>
  <link rel="preload" href="/hero.jpg" as="image" />
  <link rel="preload" href="/fonts/main.woff2" as="font" crossOrigin />
</Head>

FID (first input delay) - < 100ms

  • minimize JS
  • code splitting
  • defer non-critical JS
// dynamic imports
const HeavyComponent = dynamic(() => import('./HeavyComponent'), {
  loading: () => <p>loading...</p>,
});

CLS (cumulative layout shift) - < 0.1

  • set image dimensions
  • reserve space for ads
  • avoid inserting content above existing content

monitoring

# lighthouse
npm install -g lighthouse
lighthouse https://yoursite.com --view

# core web vitals
npm install web-vitals

import { getCLS, getFID, getLCP } from 'web-vitals';

getCLS(console.log);
getFID(console.log);
getLCP(console.log);

quick wins

  • add meta descriptions to all pages
  • create XML sitemap
  • implement structured data
  • optimize images
  • enable compression
  • set up HTTPS
  • fix broken links
  • improve mobile experience
  • reduce page load time
  • create quality content regularly

common mistakes

  • duplicate content
  • slow loading times
  • poor mobile experience
  • missing meta descriptions
  • broken internal links
  • no sitemap
  • ignoring analytics
  • keyword stuffing
  • thin content
  • no HTTPS

links